Repository: optuna/optuna

Issue:
AllenNLP integration: Can't load configurations that expect ext_vars from environment variables
Optuna's AllenNLP integration evaluates the jsonnet config that you give it, passing in the trial parameters as the `ext_vars`. AllenNLP's default behavior when loading a configuration is a little different, in that it also passes through environment variables: 

https://github.com/allenai/allennlp/blob/4de68a42a0df89df01a999ee48f361f24c8c19d4/allennlp/common/params.py#L488

So if you rely on specifying e.g. a path to a file as an environment variable, that configuration won't load as expected when using the integration

## Expected behavior

Optuna should pass environment variables through when the jsonnet is interpreted. Ideally, it'd just use the AllenNLP config loading code directly so that all behavior is identical, just overriding whatever the trial needs to override.

## Environment

- Optuna version: 1.5.0
- Python version: 3.6
- OS: Ubuntu 18.04.4
- (Optional) Other libraries and their versions: AllenNLP 1.0

## Error messages, stack traces, or logs

You'll get `RuntimeError: undefined external variable` during the interpretation of the jsonnet here:

https://github.com/optuna/optuna/blob/9f5c5cf5c36df1628567f0b4a9873892d148ad40/optuna/integration/allennlp.py#L108

## Steps to reproduce

1. Use a AllenNLP configuration that expects to be able to use an environment variable to fill an extVar


Discussion:

Resolving patch:
diff --git a/optuna/integration/allennlp.py b/optuna/integration/allennlp.py
--- a/optuna/integration/allennlp.py
+++ b/optuna/integration/allennlp.py
@@ -57,6 +57,13 @@ class AllenNLPExecutor(object):
     `config file <https://github.com/optuna/optuna/blob/master/
     examples/allennlp/classifier.jsonnet>`_.
 
+    .. note::
+        In :class:`~optuna.integration.AllenNLPExecutor`,
+        you can pass parameters to AllenNLP by either defining a search space using
+        Optuna suggest methods or setting environment variables just like AllenNLP CLI.
+        If a value is set in both a search space in Optuna and the environment variables,
+        the executor will use the value specified in the search space in Optuna.
+
     Args:
         trial:
             A :class:`~optuna.trial.Trial` corresponding to the current evaluation
@@ -100,17 +107,29 @@ def __init__(
             self._include_package = include_package
 
     def _build_params(self) -> Dict[str, Any]:
-        """Create a dict of params for AllenNLP."""
-        # _build_params is based on allentune's train_func.
-        # https://github.com/allenai/allentune/blob/master/allentune/modules/allennlp_runner.py#L34-L65
-        for key, value in self._params.items():
-            self._params[key] = str(value)
-        _params = json.loads(_jsonnet.evaluate_file(self._config_file, ext_vars=self._params))
-
-        # _params contains a list of string or string as value values.
+        """Create a dict of params for AllenNLP.
+
+        _build_params is based on allentune's train_func.
+        For more detail, please refer to
+        https://github.com/allenai/allentune/blob/master/allentune/modules/allennlp_runner.py#L34-L65
+
+        """
+        params = self._environment_variables()
+        params.update({key: str(value) for key, value in self._params.items()})
+
+        allennlp_params = json.loads(_jsonnet.evaluate_file(self._config_file, ext_vars=params))
+        # allennlp_params contains a list of string or string as value values.
         # Some params couldn't be casted correctly and
         # infer_and_cast converts them into desired values.
-        return allennlp.common.params.infer_and_cast(_params)
+        return allennlp.common.params.infer_and_cast(allennlp_params)
+
+    @staticmethod
+    def _is_encodable(value: str) -> bool:
+        # https://github.com/allenai/allennlp/blob/master/allennlp/common/params.py#L77-L85
+        return (value == "") or (value.encode("utf-8", "ignore") != b"")
+
+    def _environment_variables(self) -> Dict[str, str]:
+        return {key: value for key, value in os.environ.items() if self._is_encodable(value)}
 
     def run(self) -> float:
         """Train a model using AllenNLP."""


Test patch:
diff --git a/tests/integration_tests/allennlp_tests/example_with_environment_variables.jsonnet b/tests/integration_tests/allennlp_tests/example_with_environment_variables.jsonnet
new file mode 100644
--- /dev/null
+++ b/tests/integration_tests/allennlp_tests/example_with_environment_variables.jsonnet
@@ -0,0 +1,64 @@
+local DROPOUT = std.extVar('DROPOUT');
+local LEARNING_RATE = std.extVar('LEARNING_RATE');
+
+
+{
+  dataset_reader: {
+    type: 'sequence_tagging',
+    word_tag_delimiter: '/',
+    token_indexers: {
+      tokens: {
+        type: 'tiny_single_id',
+        lowercase_tokens: true,
+      },
+      token_characters: {
+        type: 'characters',
+      },
+    },
+  },
+  train_data_path: std.extVar('TRAIN_PATH'),
+  validation_data_path: std.extVar('VALID_PATH'),
+  model: {
+    type: 'simple_tagger',
+    text_field_embedder: {
+      token_embedders: {
+        tokens: {
+          type: 'embedding',
+          embedding_dim: 5,
+        },
+        token_characters: {
+          type: 'character_encoding',
+          embedding: {
+            embedding_dim: 4,
+          },
+          encoder: {
+            type: 'cnn',
+            embedding_dim: 4,
+            num_filters: 5,
+            ngram_filter_sizes: [3],
+          },
+          dropout: DROPOUT,
+        },
+      },
+    },
+    encoder: {
+      type: 'lstm',
+      input_size: 10,
+      hidden_size: 10,
+      num_layers: 2,
+      dropout: 0,
+      bidirectional: true,
+    },
+  },
+  iterator: {
+    type: 'basic',
+    batch_size: 32,
+  },
+  trainer: {
+    optimizer: 'adam',
+    lr: LEARNING_RATE,
+    num_epochs: 1,
+    patience: 10,
+    cuda_device: -1,
+  },
+}
diff --git a/tests/integration_tests/allennlp_tests/test_allennlp.py b/tests/integration_tests/allennlp_tests/test_allennlp.py
--- a/tests/integration_tests/allennlp_tests/test_allennlp.py
+++ b/tests/integration_tests/allennlp_tests/test_allennlp.py
@@ -1,5 +1,5 @@
 import json
-import os.path
+import os
 import tempfile
 
 import _jsonnet
@@ -8,7 +8,7 @@
 import optuna
 
 
-def test__set_param() -> None:
+def test_build_params() -> None:
     study = optuna.create_study(direction="maximize")
     trial = optuna.trial.Trial(study, study._storage.create_new_trial(study._study_id))
     trial.suggest_uniform("LEARNING_RATE", 1e-2, 1e-1)
@@ -23,6 +23,52 @@ def test__set_param() -> None:
     assert params["model"]["hidden_size"] == [100, 200, 300]
 
 
+def test_build_params_overwriting_environment_variable() -> None:
+    study = optuna.create_study(direction="maximize")
+    trial = optuna.trial.Trial(study, study._storage.create_new_trial(study._study_id))
+    trial.suggest_uniform("LEARNING_RATE", 1e-2, 1e-1)
+    trial.suggest_uniform("DROPOUT", 0.0, 0.5)
+    os.environ["TRAIN_PATH"] = "tests/integration_tests/allennlp_tests/sentences.train"
+    os.environ["VALID_PATH"] = "tests/integration_tests/allennlp_tests/sentences.valid"
+    executor = optuna.integration.AllenNLPExecutor(
+        trial,
+        "tests/integration_tests/allennlp_tests/example_with_environment_variables.jsonnet",
+        "test",
+    )
+    params = executor._build_params()
+    os.environ.pop("TRAIN_PATH")
+    os.environ.pop("VALID_PATH")
+    assert params["train_data_path"] == "tests/integration_tests/allennlp_tests/sentences.train"
+    assert (
+        params["validation_data_path"] == "tests/integration_tests/allennlp_tests/sentences.valid"
+    )
+
+
+def test_build_params_when_optuna_and_environment_variable_both_exist() -> None:
+    study = optuna.create_study(direction="maximize")
+    trial = optuna.trial.Trial(study, study._storage.create_new_trial(study._study_id))
+    trial.suggest_uniform("LEARNING_RATE", 1e-2, 1e-2)
+    os.environ["TRAIN_PATH"] = "tests/integration_tests/allennlp_tests/sentences.train"
+    os.environ["VALID_PATH"] = "tests/integration_tests/allennlp_tests/sentences.valid"
+    os.environ["LEARNING_RATE"] = "1e-3"
+    os.environ["DROPOUT"] = "0.0"
+    executor = optuna.integration.AllenNLPExecutor(
+        trial,
+        "tests/integration_tests/allennlp_tests/example_with_environment_variables.jsonnet",
+        "test",
+    )
+    params = executor._build_params()
+    os.environ.pop("TRAIN_PATH")
+    os.environ.pop("VALID_PATH")
+    os.environ.pop("LEARNING_RATE")
+    os.environ.pop("DROPOUT")
+
+    # Optuna trial overwrites a parameter specified by environment variable
+    assert params["trainer"]["lr"] == 1e-2
+    path = params["model"]["text_field_embedder"]["token_embedders"]["token_characters"]["dropout"]
+    assert path == 0.0
+
+
 def test_missing_config_file() -> None:
     study = optuna.create_study(direction="maximize")
     trial = optuna.trial.Trial(study, study._storage.create_new_trial(study._study_id))