Repository: apache/airflow

Issue:
Import connections/variables from a file
Hello,

We have a new local file system secret backend that allows us to load connections and variables from files. We support 3 file formats: YAML, JSON, ENV.
https://airflow.readthedocs.io/en/latest/howto/use-alternative-secrets-backend.html#local-filesystem-secrets-backend
https://github.com/apache/airflow/blob/master/airflow/secrets/local_filesystem.py
I think we can use the same code to create new commands for the CLI that allow us o import connections or variables. 

Best regards,
Kamil


Discussion:
>Hey mik-laj,
> 
> What should happen to existing connections? should we override them?/let them be untouched?/ask for user input?

@utkarsharma2 By default, the operation should be canceled, but with an additional option it should be possible to change this behavior ``--overwrite-existing``.

Hey @mik-laj ,

There is already a feature to import variables - `airflow variables import file`, however, this command only accepts the JSON file. Should we augment, existing command?

Also, I'll add --overwrite-existing flag.
I thought about the extra option for a longer time and I don't think this will cover all cases.

I think, we need `conflict-disposition` It should specifies the action that occurs if the variables/connection already exists. 
We should support the following values are supported:
`OVERWRITE`: If the table variable/connection already exists, we should overwrites the data.
`IGNORE`: If the table variable/connection already exists, we should skip this connection/variable.
`RESTRICT`: If the variable/connection already exists, we should raise an exception.

The default value should be RESTRICT. Each command invocation should be atomic and only occurs if all changes is able to complete successfully (one transaction).

Example:
```
airflow connection import /files/variables.json --conflict-disposition=overwrite
airflow connection import /files/variables.json --conflict-disposition=ignore
airflow connection import /files/variables.json --conflict-disposition=restrict
```


We should handle reading values from a standard input. `airflow connection import -` should read value from stdin instead file. 
@mik-laj is there any updates on this feature? I tried to run `airflow connections import $JSON_FILE` but it didn't work :(
Here is PR:https://github.com/apache/airflow/pull/9907 Feel free to continue it
I'm looking for a good first issue to work on.

Is this a correct account of the feature request?

> When using the airflow command line interface tool, I want to be able to import connections to databases or other services using a file that specifies the details associated with the connection.

Is there anything else that I should understand with respect to the request?
That sounds correct to me.
@chutcheson  Yes. We just want to import connections from a file.
I started a first crack at this issue a few days ago, apologies for not posting sooner. I'll open a PR for it shortly. @chutcheson, would be great to get your feedback since you're interested in this issue as well!

Resolving patch:
diff --git a/airflow/cli/commands/connection_command.py b/airflow/cli/commands/connection_command.py
--- a/airflow/cli/commands/connection_command.py
+++ b/airflow/cli/commands/connection_command.py
@@ -28,7 +28,7 @@
 from airflow.exceptions import AirflowNotFoundException
 from airflow.hooks.base import BaseHook
 from airflow.models import Connection
-from airflow.secrets.local_filesystem import _create_connection, load_connections_dict
+from airflow.secrets.local_filesystem import load_connections_dict
 from airflow.utils import cli as cli_utils, yaml
 from airflow.utils.cli import suppress_logs_and_warning
 from airflow.utils.session import create_session
@@ -238,7 +238,7 @@ def connections_delete(args):
 
 @cli_utils.action_logging
 def connections_import(args):
-    """Imports connections from a given file"""
+    """Imports connections from a file"""
     if os.path.exists(args.file):
         _import_helper(args.file)
     else:
@@ -246,31 +246,14 @@ def connections_import(args):
 
 
 def _import_helper(file_path):
-    """Helps import connections from a file"""
+    """Load connections from a file and save them to the DB. On collision, skip."""
     connections_dict = load_connections_dict(file_path)
     with create_session() as session:
-        for conn_id, conn_values in connections_dict.items():
+        for conn_id, conn in connections_dict.items():
             if session.query(Connection).filter(Connection.conn_id == conn_id).first():
                 print(f'Could not import connection {conn_id}: connection already exists.')
                 continue
 
-            allowed_fields = [
-                'extra',
-                'description',
-                'conn_id',
-                'login',
-                'conn_type',
-                'host',
-                'password',
-                'schema',
-                'port',
-                'uri',
-                'extra_dejson',
-            ]
-            filtered_connection_values = {
-                key: value for key, value in conn_values.items() if key in allowed_fields
-            }
-            connection = _create_connection(conn_id, filtered_connection_values)
-            session.add(connection)
+            session.add(conn)
             session.commit()
             print(f'Imported connection {conn_id}')
diff --git a/airflow/models/connection.py b/airflow/models/connection.py
--- a/airflow/models/connection.py
+++ b/airflow/models/connection.py
@@ -19,7 +19,7 @@
 import json
 import warnings
 from json import JSONDecodeError
-from typing import Dict, Optional
+from typing import Dict, Optional, Union
 from urllib.parse import parse_qsl, quote, unquote, urlencode, urlparse
 
 from sqlalchemy import Boolean, Column, Integer, String, Text
@@ -117,12 +117,14 @@ def __init__(  # pylint: disable=too-many-arguments
         password: Optional[str] = None,
         schema: Optional[str] = None,
         port: Optional[int] = None,
-        extra: Optional[str] = None,
+        extra: Optional[Union[str, dict]] = None,
         uri: Optional[str] = None,
     ):
         super().__init__()
         self.conn_id = conn_id
         self.description = description
+        if extra and not isinstance(extra, str):
+            extra = json.dumps(extra)
         if uri and (  # pylint: disable=too-many-boolean-expressions
             conn_type or host or login or password or schema or port or extra
         ):


Test patch:
diff --git a/tests/cli/commands/test_connection_command.py b/tests/cli/commands/test_connection_command.py
--- a/tests/cli/commands/test_connection_command.py
+++ b/tests/cli/commands/test_connection_command.py
@@ -758,9 +758,9 @@ def test_cli_connections_import_should_return_error_if_file_format_is_invalid(
         ):
             connection_command.connections_import(self.parser.parse_args(["connections", "import", filepath]))
 
-    @mock.patch('airflow.cli.commands.connection_command.load_connections_dict')
+    @mock.patch('airflow.secrets.local_filesystem._parse_secret_file')
     @mock.patch('os.path.exists')
-    def test_cli_connections_import_should_load_connections(self, mock_exists, mock_load_connections_dict):
+    def test_cli_connections_import_should_load_connections(self, mock_exists, mock_parse_secret_file):
         mock_exists.return_value = True
 
         # Sample connections to import
@@ -769,26 +769,26 @@ def test_cli_connections_import_should_load_connections(self, mock_exists, mock_
                 "conn_type": "postgres",
                 "description": "new0 description",
                 "host": "host",
-                "is_encrypted": False,
-                "is_extra_encrypted": False,
                 "login": "airflow",
+                "password": "password",
                 "port": 5432,
                 "schema": "airflow",
+                "extra": "test",
             },
             "new1": {
                 "conn_type": "mysql",
                 "description": "new1 description",
                 "host": "host",
-                "is_encrypted": False,
-                "is_extra_encrypted": False,
                 "login": "airflow",
+                "password": "password",
                 "port": 3306,
                 "schema": "airflow",
+                "extra": "test",
             },
         }
 
-        # We're not testing the behavior of load_connections_dict, assume successfully reads JSON, YAML or env
-        mock_load_connections_dict.return_value = expected_connections
+        # We're not testing the behavior of _parse_secret_file, assume it successfully reads JSON, YAML or env
+        mock_parse_secret_file.return_value = expected_connections
 
         connection_command.connections_import(
             self.parser.parse_args(["connections", "import", 'sample.json'])
@@ -799,14 +799,15 @@ def test_cli_connections_import_should_load_connections(self, mock_exists, mock_
             current_conns = session.query(Connection).all()
 
             comparable_attrs = [
+                "conn_id",
                 "conn_type",
                 "description",
                 "host",
-                "is_encrypted",
-                "is_extra_encrypted",
                 "login",
+                "password",
                 "port",
                 "schema",
+                "extra",
             ]
 
             current_conns_as_dicts = {
@@ -816,80 +817,81 @@ def test_cli_connections_import_should_load_connections(self, mock_exists, mock_
             assert expected_connections == current_conns_as_dicts
 
     @provide_session
-    @mock.patch('airflow.cli.commands.connection_command.load_connections_dict')
+    @mock.patch('airflow.secrets.local_filesystem._parse_secret_file')
     @mock.patch('os.path.exists')
     def test_cli_connections_import_should_not_overwrite_existing_connections(
-        self, mock_exists, mock_load_connections_dict, session=None
+        self, mock_exists, mock_parse_secret_file, session=None
     ):
         mock_exists.return_value = True
 
-        # Add a pre-existing connection "new1"
+        # Add a pre-existing connection "new3"
         merge_conn(
             Connection(
-                conn_id="new1",
+                conn_id="new3",
                 conn_type="mysql",
-                description="mysql description",
+                description="original description",
                 host="mysql",
                 login="root",
-                password="",
+                password="password",
                 schema="airflow",
             ),
             session=session,
         )
 
-        # Sample connections to import, including a collision with "new1"
+        # Sample connections to import, including a collision with "new3"
         expected_connections = {
-            "new0": {
+            "new2": {
                 "conn_type": "postgres",
-                "description": "new0 description",
+                "description": "new2 description",
                 "host": "host",
-                "is_encrypted": False,
-                "is_extra_encrypted": False,
                 "login": "airflow",
+                "password": "password",
                 "port": 5432,
                 "schema": "airflow",
+                "extra": "test",
             },
-            "new1": {
+            "new3": {
                 "conn_type": "mysql",
-                "description": "new1 description",
+                "description": "updated description",
                 "host": "host",
-                "is_encrypted": False,
-                "is_extra_encrypted": False,
                 "login": "airflow",
+                "password": "new password",
                 "port": 3306,
                 "schema": "airflow",
+                "extra": "test",
             },
         }
 
-        # We're not testing the behavior of load_connections_dict, assume successfully reads JSON, YAML or env
-        mock_load_connections_dict.return_value = expected_connections
+        # We're not testing the behavior of _parse_secret_file, assume it successfully reads JSON, YAML or env
+        mock_parse_secret_file.return_value = expected_connections
 
         with redirect_stdout(io.StringIO()) as stdout:
             connection_command.connections_import(
                 self.parser.parse_args(["connections", "import", 'sample.json'])
             )
 
-            assert 'Could not import connection new1: connection already exists.' in stdout.getvalue()
+            assert 'Could not import connection new3: connection already exists.' in stdout.getvalue()
 
         # Verify that the imported connections match the expected, sample connections
         current_conns = session.query(Connection).all()
 
         comparable_attrs = [
+            "conn_id",
             "conn_type",
             "description",
             "host",
-            "is_encrypted",
-            "is_extra_encrypted",
             "login",
+            "password",
             "port",
             "schema",
+            "extra",
         ]
 
         current_conns_as_dicts = {
             current_conn.conn_id: {attr: getattr(current_conn, attr) for attr in comparable_attrs}
             for current_conn in current_conns
         }
-        assert current_conns_as_dicts['new0'] == expected_connections['new0']
+        assert current_conns_as_dicts['new2'] == expected_connections['new2']
 
         # The existing connection's description should not have changed
-        assert current_conns_as_dicts['new1']['description'] == 'new1 description'
+        assert current_conns_as_dicts['new3']['description'] == 'original description'