Repository: amundsen-io/amundsen

Issue:
[Feast Extractor] Feast client needs a valid Serving URL, instead default localhost
<!--- Provide a general summary of the issue in the Title above -->
<!--- Look through existing open and closed issues to see if someone has reported the issue before -->
Feast extractor is failing when the Feast Serving Service is not the default `localhost:6566`. It is because the Feast client tests the connection with Serving URL when starting, even if we'll not use the Online service.

## Expected Behavior
- Use the Feast extractor even when Feast is running in another host than `localhost`

## Current Behavior
- Running the Feast Extractor with a Feast not running in the `localhost` causes the following error:
```python
[2021-02-24 22:59:34,163] {job.py:59} INFO - Launching a job
[2021-02-24 22:59:34,165] {task.py:53} INFO - Running a task
[2021-02-24 22:59:44,191] {file_system_neo4j_csv_loader.py:88} INFO - Deleting directory /var/tmp/amundsen/table_metadata/relationships/
[2021-02-24 22:59:44,192] {file_system_neo4j_csv_loader.py:88} INFO - Deleting directory /var/tmp/amundsen/table_metadata/nodes/
[2021-02-24 22:59:44,192] {taskinstance.py:1455} ERROR - Connection timed out while attempting to connect to localhost:6566
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/feast/grpc/grpc.py", line 48, in create_grpc_channel
    grpc.channel_ready_future(channel).result(timeout=timeout)
  File "/home/airflow/.local/lib/python3.6/site-packages/grpc/_utilities.py", line 140, in result
    self._block(timeout)
  File "/home/airflow/.local/lib/python3.6/site-packages/grpc/_utilities.py", line 86, in _block
    raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError
```

## Possible Solution
- Add `serving_url` parameter to the client creation, for example:
 ```python
class FeastExtractor(Extractor):
    """
    Extracts feature tables from Feast Core service. Since Feast is
    a metadata store (and not the database itself), it maps the
    following atributes:
     * a database is name of feast project
     * table name is a name of the feature table
     * columns are features stored in the feature table
    """

    FEAST_SERVICE_CONFIG_KEY = "instance_name"
    FEAST_ENDPOINT_CONFIG_KEY = "endpoint"
    FEAST_SERVING_ENDPOINT_CONFIG_KEY="serving_endpoint"
    DESCRIBE_FEATURE_TABLES = "describe_feature_tables"
    DEFAULT_CONFIG = ConfigFactory.from_dict(
        {FEAST_SERVICE_CONFIG_KEY: "main", DESCRIBE_FEATURE_TABLES: True}
    )

    def init(self, conf: ConfigTree) -> None:
        conf = conf.with_fallback(FeastExtractor.DEFAULT_CONFIG)
        self._feast_service = conf.get_string(FeastExtractor.FEAST_SERVICE_CONFIG_KEY)
        self._describe_feature_tables = conf.get_bool(
            FeastExtractor.DESCRIBE_FEATURE_TABLES
        )
        self._client = Client(
            core_url=conf.get_string(FeastExtractor.FEAST_ENDPOINT_CONFIG_KEY),
            serving_url=conf.get_string(FeastExtractor.FEAST_SERVING_ENDPOINT_CONFIG_KEY),
        )
        self._extract_iter: Union[None, Iterator] = None
```

- Or receive a dictionary with Feast client options, as exemplified in the [Feast notebook example](https://github.com/feast-dev/feast/blob/master/examples/minimal/minimal_ride_hailing.ipynb)

## Steps to Reproduce
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
1. Run Feast Extractor Sample with a Feast Client running in a host other than `localhost`

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Amunsen version used: amundsen-databuilder = "^4.1.0"
* Feast: feast==0.9.2 



Discussion:
cool, could you create a pr to fix that? thanks
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@amommendes hey, would still love this contribution if you're able to subit as a PR!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

hey @dorianj, now I´m able to submit this PR.. However, it is possible that is already out of date, Feast changed a lot from 0.9 to 0.10
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.


Resolving patch:
diff --git a/databuilder/databuilder/extractor/feast_extractor.py b/databuilder/databuilder/extractor/feast_extractor.py
--- a/databuilder/databuilder/extractor/feast_extractor.py
+++ b/databuilder/databuilder/extractor/feast_extractor.py
@@ -26,6 +26,7 @@ class FeastExtractor(Extractor):
 
     FEAST_SERVICE_CONFIG_KEY = "instance_name"
     FEAST_ENDPOINT_CONFIG_KEY = "endpoint"
+    FEAST_SERVING_ENDPOINT_CONFIG_KEY = "serving_endpoint"
     DESCRIBE_FEATURE_TABLES = "describe_feature_tables"
     DEFAULT_CONFIG = ConfigFactory.from_dict(
         {FEAST_SERVICE_CONFIG_KEY: "main", DESCRIBE_FEATURE_TABLES: True}
@@ -38,7 +39,8 @@ def init(self, conf: ConfigTree) -> None:
             FeastExtractor.DESCRIBE_FEATURE_TABLES
         )
         self._client = Client(
-            core_url=conf.get_string(FeastExtractor.FEAST_ENDPOINT_CONFIG_KEY)
+            core_url=conf.get_string(FeastExtractor.FEAST_ENDPOINT_CONFIG_KEY),
+            serving_url=conf.get_string(FeastExtractor.FEAST_SERVING_ENDPOINT_CONFIG_KEY),
         )
         self._extract_iter: Union[None, Iterator] = None
 
diff --git a/databuilder/example/scripts/sample_feast_loader.py b/databuilder/example/scripts/sample_feast_loader.py
--- a/databuilder/example/scripts/sample_feast_loader.py
+++ b/databuilder/example/scripts/sample_feast_loader.py
@@ -28,8 +28,9 @@
 from databuilder.task.task import DefaultTask
 
 feast_endpoint = sys.argv[1]
-neo4j_endpoint = sys.argv[2]
-es_url = sys.argv[3]
+feast_serving_endpoint = sys.argv[2]
+neo4j_endpoint = sys.argv[3]
+es_url = sys.argv[4]
 es = Elasticsearch([es_url])
 
 neo4j_user = "neo4j"
@@ -44,6 +45,7 @@ def create_feast_job_config():
     job_config = ConfigFactory.from_dict(
         {
             f"extractor.feast.{FeastExtractor.FEAST_ENDPOINT_CONFIG_KEY}": feast_endpoint,
+            f'extractor.feast.{FeastExtractor.FEAST_SERVING_ENDPOINT_CONFIG_KEY}': feast_serving_endpoint,
             f"loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.NODE_DIR_PATH}": node_files_folder,
             f"loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.RELATION_DIR_PATH}": relationship_files_folder,
             f"publisher.neo4j.{neo4j_csv_publisher.NODE_FILES_DIR}": node_files_folder,


Test patch:
diff --git a/databuilder/tests/unit/extractor/test_feast_extractor.py b/databuilder/tests/unit/extractor/test_feast_extractor.py
--- a/databuilder/tests/unit/extractor/test_feast_extractor.py
+++ b/databuilder/tests/unit/extractor/test_feast_extractor.py
@@ -166,6 +166,7 @@ def test_feature_table_extraction_with_description_stream(self) -> None:
     def _init_extractor(self, programmatic_description_enabled: bool = True) -> None:
         conf = {
             f'extractor.feast.{FeastExtractor.FEAST_ENDPOINT_CONFIG_KEY}': 'feast-core.example.com:6565',
+            f'extractor.feast.{FeastExtractor.FEAST_SERVING_ENDPOINT_CONFIG_KEY}': 'feast-core.example.com:6566',
             f'extractor.feast.{FeastExtractor.FEAST_SERVICE_CONFIG_KEY}': 'unittest-feast-instance',
             f'extractor.feast.{FeastExtractor.DESCRIBE_FEATURE_TABLES}': programmatic_description_enabled,
         }