internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: feast-dev/feast

Issue:
Redshift does not exit long running query
https://github.com/feast-dev/feast/blob/fd255cae7153cd44432f172b1e5c4738e4a7d583/sdk/python/feast/infra/utils/aws_utils.py#L88-L91

There is no timeout on this line of code, which means it is possible for queries to run indefinitely if they don't raise an exception. I've personally run into a query that has run for 25 minutes with no response.


Discussion:

Resolving patch:
diff --git a/sdk/python/feast/infra/utils/aws_utils.py b/sdk/python/feast/infra/utils/aws_utils.py
--- a/sdk/python/feast/infra/utils/aws_utils.py
+++ b/sdk/python/feast/infra/utils/aws_utils.py
@@ -7,7 +7,13 @@
 import pandas as pd
 import pyarrow as pa
 import pyarrow.parquet as pq
-from tenacity import retry, retry_if_exception_type, wait_exponential
+from tenacity import (
+    retry,
+    retry_if_exception_type,
+    stop_after_attempt,
+    stop_after_delay,
+    wait_exponential,
+)
 
 from feast.errors import RedshiftCredentialsError, RedshiftQueryError
 from feast.type_map import pa_to_redshift_value_type
@@ -53,6 +59,7 @@ def get_bucket_and_key(s3_path: str) -> Tuple[str, str]:
 @retry(
     wait=wait_exponential(multiplier=1, max=4),
     retry=retry_if_exception_type(ConnectionClosedError),
+    stop=stop_after_attempt(5),
 )
 def execute_redshift_statement_async(
     redshift_data_client, cluster_id: str, database: str, user: str, query: str
@@ -88,6 +95,7 @@ class RedshiftStatementNotFinishedError(Exception):
 @retry(
     wait=wait_exponential(multiplier=1, max=30),
     retry=retry_if_exception_type(RedshiftStatementNotFinishedError),
+    stop=stop_after_delay(300),  # 300 seconds
 )
 def wait_for_redshift_statement(redshift_data_client, statement: dict) -> None:
     """Waits for the Redshift statement to finish. Raises RedshiftQueryError if the statement didn't succeed.


Test patch:
Repository: feast-dev/feast

Issue:
feast alpha enable command messes up feature_store.yaml
## Expected Behavior

Calling `feast alpha enable python_feature_server` should add
```
flags:
  alpha_features: true
  python_feature_server: true
```
to `feature_store.yaml`.

## Current Behavior

Instead, it changes
```
project: sharing_sculpin
registry: data/registry.db
provider: local
online_store:
    path: data/online_store.db
```
to
```
project: sharing_sculpin
provider: local
online_store:
  path: data/online_store.db
offline_store: {}
flags:
  alpha_features: true
  python_feature_server: true
```
This still works due to the Pydantic defaults, but can be very confusing for users.

Edit: in some cases, it actually doesn't preserve the same effect. For example, the command changes
```
project: sharing_sculpin
provider: local
online_store:
  type: redis
offline_store: {}
flags:
  alpha_features: true
  python_feature_server: true
```
to
```
project: sharing_sculpin
provider: local
online_store: {}
offline_store: {}
flags:
  alpha_features: true
  python_feature_server: true
```
which is incorrect since the `online_store` no longer has type `redis`.

## Steps to reproduce

Call `feast init` to create a new `feature_repo`. Then call `feast alpha enable python_feature_server`, and inspect `feature_store.yaml` before and after this command.

### Specifications

- Version:
- Platform:
- Subsystem:

## Possible Solution



Discussion:

Resolving patch:
diff --git a/sdk/python/feast/repo_config.py b/sdk/python/feast/repo_config.py
--- a/sdk/python/feast/repo_config.py
+++ b/sdk/python/feast/repo_config.py
@@ -277,14 +277,7 @@ def write_to_path(self, repo_path: Path):
         config_path = repo_path / "feature_store.yaml"
         with open(config_path, mode="w") as f:
             yaml.dump(
-                yaml.safe_load(
-                    self.json(
-                        exclude={"repo_path"},
-                        exclude_none=True,
-                        exclude_unset=True,
-                        exclude_defaults=True,
-                    )
-                ),
+                yaml.safe_load(self.json(exclude={"repo_path"}, exclude_unset=True,)),
                 f,
                 sort_keys=False,
             )


Test patch:
Repository: feast-dev/feast

Issue:
Solve feast conflict dependencies for [gcp]
## Expected Behavior 
`pip-compile` should run without error and result in a nice lock files of the libraries to install

## Current Behavior
`pip-compile` will find conflict in the current feast[gcp] dependencies. Did not try for [aws] or [redis]
<img width="1774" alt="image" src="https://user-images.githubusercontent.com/18557047/135331637-5c3d47ad-ebe0-4a27-b335-93617675027f.png">

## Steps to reproduce
```bash
echo "-e  sdk/python[gcp]" > sdk/python/requirements.txt
pip-compile --dry-run sdk/python/requirements.txt
```
<img width="1244" alt="image" src="https://user-images.githubusercontent.com/18557047/135332916-c368ca80-3276-40ab-a3bd-42c48d52c2e9.png">

### Specifications

- Version:
- Platform:
- Subsystem:

## Possible Solution



Discussion:

Resolving patch:
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -69,7 +69,7 @@
     "google-cloud-bigquery>=2.28.1",
     "google-cloud-bigquery-storage >= 2.0.0",
     "google-cloud-datastore>=2.1.*",
-    "google-cloud-storage>=1.34.*",
+    "google-cloud-storage>=1.34.*,<1.41",
     "google-cloud-core==1.4.*",
 ]
 
@@ -115,7 +115,7 @@
     "google-cloud-bigquery>=2.28.1",
     "google-cloud-bigquery-storage >= 2.0.0",
     "google-cloud-datastore>=2.1.*",
-    "google-cloud-storage>=1.20.*",
+    "google-cloud-storage>=1.20.*,<1.41",
     "google-cloud-core==1.4.*",
     "redis-py-cluster==2.1.2",
     "boto3==1.17.*",


Test patch:
Repository: feast-dev/feast

Issue:
Dataset location harcoded to US when using BigQueryOfflineStore.get_historical_features()
## Expected Behavior 
Create the BQ dataset in the location specified in feature_store.yaml if the dataset doesn't exists
## Current Behavior
Always create the BQ dataset in the US, raising a _dataset not found_ since the dataset is not located in the same region as the one specified in _feature_store.yam_ config file
## Steps to reproduce
use an bigquery offline_store with location set to EU in feature_store.yaml

### Specifications
https://github.com/feast-dev/feast/blob/ec4c02bb149bd8462697644ea20e2660ddfba81a/sdk/python/feast/infra/offline_stores/bigquery.py#L340
- Version: 
- Platform:
- Subsystem:

## Possible Solution
update feast.infra.offline_stores.bigquery._get_table_reference_for_new_entity() by adding a new _dataset_location_ arg
Dataset location harcoded to US when using BigQueryOfflineStore.get_historical_features()
## Expected Behavior 
Create the BQ dataset in the location specified in feature_store.yaml if the dataset doesn't exists
## Current Behavior
Always create the BQ dataset in the US, raising a _dataset not found_ since the dataset is not located in the same region as the one specified in _feature_store.yam_ config file
## Steps to reproduce
use an bigquery offline_store with location set to EU in feature_store.yaml

### Specifications
https://github.com/feast-dev/feast/blob/ec4c02bb149bd8462697644ea20e2660ddfba81a/sdk/python/feast/infra/offline_stores/bigquery.py#L340
- Version: 
- Platform:
- Subsystem:

## Possible Solution
update feast.infra.offline_stores.bigquery._get_table_reference_for_new_entity() by adding a new _dataset_location_ arg


Discussion:
I'll do a PR
I'll do a PR

Resolving patch:
diff --git a/sdk/python/feast/infra/offline_stores/bigquery.py b/sdk/python/feast/infra/offline_stores/bigquery.py
--- a/sdk/python/feast/infra/offline_stores/bigquery.py
+++ b/sdk/python/feast/infra/offline_stores/bigquery.py
@@ -136,7 +136,10 @@ def get_historical_features(
         assert isinstance(config.offline_store, BigQueryOfflineStoreConfig)
 
         table_reference = _get_table_reference_for_new_entity(
-            client, client.project, config.offline_store.dataset
+            client,
+            client.project,
+            config.offline_store.dataset,
+            config.offline_store.location,
         )
 
         @contextlib.contextmanager
@@ -340,13 +343,16 @@ def _wait_until_done(bq_job):
 
 
 def _get_table_reference_for_new_entity(
-    client: Client, dataset_project: str, dataset_name: str
+    client: Client,
+    dataset_project: str,
+    dataset_name: str,
+    dataset_location: Optional[str],
 ) -> str:
     """Gets the table_id for the new entity to be uploaded."""
 
     # First create the BigQuery dataset if it doesn't exist
     dataset = bigquery.Dataset(f"{dataset_project}.{dataset_name}")
-    dataset.location = "US"
+    dataset.location = dataset_location if dataset_location else "US"
 
     try:
         client.get_dataset(dataset)


Test patch:
Repository: feast-dev/feast

Issue:
Redis Cluster materialization error
## Expected Behavior 
Materialization for Redis Cluster works 

## Current Behavior
During materialization Feast use Redis client instead of RedisCluster client

## Steps to reproduce
configure redis cluster 
```
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: redis
  redis_type: redis_cluster
  connection_string: "redis1:6379,redis2:6379,ssl=true,password=my_password"
```
try to materialize

### Specifications

- Version: 0.14.0
- Platform: python 3.7
- Subsystem: linux (anaconda3 docker image)

## Possible Solution
correct _get_client method in redis.py l. 124



Discussion:

Resolving patch:
diff --git a/sdk/python/feast/infra/online_stores/redis.py b/sdk/python/feast/infra/online_stores/redis.py
--- a/sdk/python/feast/infra/online_stores/redis.py
+++ b/sdk/python/feast/infra/online_stores/redis.py
@@ -142,7 +142,7 @@ def _get_client(self, online_store_config: RedisOnlineStoreConfig):
             startup_nodes, kwargs = self._parse_connection_string(
                 online_store_config.connection_string
             )
-            if online_store_config.type == RedisType.redis_cluster:
+            if online_store_config.redis_type == RedisType.redis_cluster:
                 kwargs["startup_nodes"] = startup_nodes
                 self._client = RedisCluster(**kwargs)
             else:


Test patch:
Repository: feast-dev/feast

Issue:
Error in requirement definition 
After installing feast in a conda environment, exporting the environment to a .yml fails. This is probably related to a missing comma at the end of this line:
https://github.com/feast-dev/feast/blob/63680bad344f55e41a281d643bef5972f2ea28da/sdk/python/setup.py#L50



Discussion:

Resolving patch:
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -47,14 +47,14 @@
     "google-api-core>=1.23.0",
     "googleapis-common-protos==1.52.*",
     "grpcio>=1.34.0",
-    "grpcio-reflection>=1.34.0"
+    "grpcio-reflection>=1.34.0",
     "Jinja2>=2.0.0",
     "jsonschema",
     "mmh3",
     "pandas>=1.0.0",
     "pandavro==1.5.*",
     "protobuf>=3.10",
-    "proto-plus",
+    "proto-plus<1.19.7",
     "pyarrow>=4.0.0",
     "pydantic>=1.0.0",
     "PyYAML>=5.4.*",
@@ -67,7 +67,6 @@
 ]
 
 GCP_REQUIRED = [
-    "proto-plus<1.19.7",
     "google-cloud-bigquery>=2.28.1",
     "google-cloud-bigquery-storage >= 2.0.0",
     "google-cloud-datastore>=2.1.*",


Test patch:
Repository: feast-dev/feast

Issue:
Make direct data ingestion into online store faster
We use the direct data ingestion capability so we can write inference data in a real-time flow and then use it immediately. We need to be able it ingest in milliseconds. Currently, the online ingestion takes a couple of seconds since it does a lookup to refresh the registry for every call to `write_to_online_store`. This blocks the new capability from being used in real-time. 

It would be great to have an `allow_cache` argument added to `write_to_online_store` so that we can do the write without needing to refresh the registry. 

Currently, I moved the registry to Postgres so the refresh is quicker, but it is still not quick enough


Discussion:
@benny-hal thanks for the issue - this makes a lot of sense. 

Resolving patch:
diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py
--- a/sdk/python/feast/feature_store.py
+++ b/sdk/python/feast/feature_store.py
@@ -888,7 +888,10 @@ def tqdm_builder(length):
 
     @log_exceptions_and_usage
     def write_to_online_store(
-        self, feature_view_name: str, df: pd.DataFrame,
+        self,
+        feature_view_name: str,
+        df: pd.DataFrame,
+        allow_registry_cache: bool = True,
     ):
         """
         ingests data directly into the Online store
@@ -899,7 +902,9 @@ def write_to_online_store(
             )
 
         # TODO: restrict this to work with online StreamFeatureViews and validate the FeatureView type
-        feature_view = self._registry.get_feature_view(feature_view_name, self.project)
+        feature_view = self._registry.get_feature_view(
+            feature_view_name, self.project, allow_cache=allow_registry_cache
+        )
         entities = []
         for entity_name in feature_view.entities:
             entities.append(self._registry.get_entity(entity_name, self.project))
diff --git a/sdk/python/feast/registry.py b/sdk/python/feast/registry.py
--- a/sdk/python/feast/registry.py
+++ b/sdk/python/feast/registry.py
@@ -519,19 +519,22 @@ def get_feature_table(self, name: str, project: str) -> FeatureTable:
                 return FeatureTable.from_proto(feature_table_proto)
         raise FeatureTableNotFoundException(name, project)
 
-    def get_feature_view(self, name: str, project: str) -> FeatureView:
+    def get_feature_view(
+        self, name: str, project: str, allow_cache: bool = False
+    ) -> FeatureView:
         """
         Retrieves a feature view.
 
         Args:
             name: Name of feature view
             project: Feast project that this feature view belongs to
+            allow_cache: Allow returning feature view from the cached registry
 
         Returns:
             Returns either the specified feature view, or raises an exception if
             none is found
         """
-        registry_proto = self._get_registry_proto()
+        registry_proto = self._get_registry_proto(allow_cache=allow_cache)
         for feature_view_proto in registry_proto.feature_views:
             if (
                 feature_view_proto.spec.name == name


Test patch:
Repository: feast-dev/feast

Issue:
requested_features are not passed to online_read() from passthrough_provider
`OnlineStore.online_read()` has a parameter `requested_features`. Similarly, `PassthroughProvider.online_read()` also has a parameter `requested_features`.
`PassthroughProvider.online_read()`  calls `OnlineStore.online_read()` but doesn't pass on the `requested_features` variable.
I am writing a new OnlineStore where I need the `requested_features` in `OnlineStore.online_read()`.

## Issue Detected 
In [feast/infra/passthrough_provider.py#L90](https://github.com/feast-dev/feast/blob/2d3cea1d1485d0af1a35a7f30abfed475b58973a/sdk/python/feast/infra/passthrough_provider.py#L90)
```
def online_read(
        self,
        config: RepoConfig,
        table: Union[FeatureTable, FeatureView],
        entity_keys: List[EntityKeyProto],
        requested_features: List[str] = None,
    ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]:
        set_usage_attribute("provider", self.__class__.__name__)
        result = self.online_store.online_read(config, table, entity_keys) # should also pass requested_features

        return result
```

- Version: 0.15.1
- Platform: all
- Subsystem: all

## Possible Solution

In [feast/infra/passthrough_provider.py#L90](https://github.com/feast-dev/feast/blob/2d3cea1d1485d0af1a35a7f30abfed475b58973a/sdk/python/feast/infra/passthrough_provider.py#L90)
`result = self.online_store.online_read(config, table, entity_keys)` 
change to 
`result = self.online_store.online_read(config, table, entity_keys, requested_features)`

Note: This won't break any of the current tests since none of the Online store implementation uses requested_features in online_read() method.

If it was as per design, can someone explain why the variable `requested_features` was not passed on? 



Discussion:
Good catch @aurobindoc , this is definitely a bug. Would you like to take a stab at fixing it? 
Sure. I will raise a PR with the fix

Resolving patch:
diff --git a/sdk/python/feast/infra/passthrough_provider.py b/sdk/python/feast/infra/passthrough_provider.py
--- a/sdk/python/feast/infra/passthrough_provider.py
+++ b/sdk/python/feast/infra/passthrough_provider.py
@@ -87,7 +87,9 @@ def online_read(
         requested_features: List[str] = None,
     ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]:
         set_usage_attribute("provider", self.__class__.__name__)
-        result = self.online_store.online_read(config, table, entity_keys)
+        result = self.online_store.online_read(
+            config, table, entity_keys, requested_features
+        )
 
         return result
 


Test patch:
Repository: feast-dev/feast

Issue:
Apply is incorrectly reporting tables to add as tables to delete
## Expected Behavior 

```
feast apply
Deploying infrastructure for ...
```

## Current Behavior
```
feast apply
Removing infrastructure for ...
```

## Steps to reproduce
Add a feature view and run feast apply.

### Specifications

- Version:
- Platform:
- Subsystem:

## Possible Solution
Luckily this looks like it is only a logging bug.  The tuple is just set in the wrong order. `extract_objects_for_apply_delete` returns keep then delete, but `apply_total_with_repo_instance` sets the variables in reverse.

```
(
        all_to_apply,
        all_to_delete,
        views_to_delete,
        views_to_keep,
    ) = extract_objects_for_apply_delete(project, registry, repo)
```

```
def extract_objects_for_apply_delete(project, registry, repo):
    ...
    return (
        all_to_apply,
        all_to_delete,
        set(
            objs_to_add[FeastObjectType.FEATURE_VIEW].union(
                objs_to_update[FeastObjectType.FEATURE_VIEW]
            )
        ),
        objs_to_delete[FeastObjectType.FEATURE_VIEW],
    )
```



Discussion:

Resolving patch:
diff --git a/sdk/python/feast/repo_operations.py b/sdk/python/feast/repo_operations.py
--- a/sdk/python/feast/repo_operations.py
+++ b/sdk/python/feast/repo_operations.py
@@ -209,8 +209,8 @@ def apply_total_with_repo_instance(
     (
         all_to_apply,
         all_to_delete,
-        views_to_delete,
         views_to_keep,
+        views_to_delete,
     ) = extract_objects_for_apply_delete(project, registry, repo)
 
     click.echo(registry_diff.to_string())
@@ -224,7 +224,7 @@ def apply_total_with_repo_instance(
 
 
 def log_infra_changes(
-    views_to_keep: List[FeatureView], views_to_delete: List[FeatureView]
+    views_to_keep: Set[FeatureView], views_to_delete: Set[FeatureView]
 ):
     from colorama import Fore, Style
 


Test patch:
Repository: feast-dev/feast

Issue:
Makefile for Running Contrib offline store tests
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Currently the contrib folder in the infra/offline_stores folder contains offline store implementations(i.e spark) that cannot be easily run against the universal integration test suite.
We would like to add a Makefile that can run contrib against the integration test suite that
s separate from the normal integration test run because we don't want to break anything in the normal github workflow.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Make a makefile
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.



Discussion:

Resolving patch:
diff --git a/sdk/python/feast/infra/offline_stores/contrib/__init__.py b/sdk/python/feast/infra/offline_stores/contrib/__init__.py
new file mode 100644
diff --git a/sdk/python/feast/infra/offline_stores/contrib/contrib_repo_configuration.py b/sdk/python/feast/infra/offline_stores/contrib/contrib_repo_configuration.py
new file mode 100644
--- /dev/null
+++ b/sdk/python/feast/infra/offline_stores/contrib/contrib_repo_configuration.py
@@ -0,0 +1,10 @@
+from tests.integration.feature_repos.integration_test_repo_config import (
+    IntegrationTestRepoConfig,
+)
+from tests.integration.feature_repos.universal.data_sources.spark_data_source_creator import (
+    SparkDataSourceCreator,
+)
+
+FULL_REPO_CONFIGS = [
+    IntegrationTestRepoConfig(offline_store_creator=SparkDataSourceCreator)
+]


Test patch:
Repository: feast-dev/feast

Issue:
Inline data sources not added to the registry fields
## Expected Behavior 
If I create a feature view with an inline data source, it should be added to the registry and discoverable via `feature_store.get_data_source`.


## Current Behavior
Only data sources defined as top level objects of the feature repo are stored and discoverable in the registry.

## Possible Solution
In registry.py, introspect all feature views and collect them to be added to the registry.



Discussion:

Resolving patch:
diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py
--- a/sdk/python/feast/feature_store.py
+++ b/sdk/python/feast/feature_store.py
@@ -560,7 +560,7 @@ def _plan(
         new_infra_proto = new_infra.to_proto()
         infra_diff = diff_infra_protos(current_infra_proto, new_infra_proto)
 
-        return (registry_diff, infra_diff, new_infra)
+        return registry_diff, infra_diff, new_infra
 
     @log_exceptions_and_usage
     def _apply_diffs(
@@ -648,16 +648,23 @@ def apply(
         ]
         odfvs_to_update = [ob for ob in objects if isinstance(ob, OnDemandFeatureView)]
         services_to_update = [ob for ob in objects if isinstance(ob, FeatureService)]
-        data_sources_to_update = [ob for ob in objects if isinstance(ob, DataSource)]
-
-        if len(entities_to_update) + len(views_to_update) + len(
-            request_views_to_update
-        ) + len(odfvs_to_update) + len(services_to_update) + len(
-            data_sources_to_update
-        ) != len(
-            objects
-        ):
-            raise ValueError("Unknown object type provided as part of apply() call")
+        data_sources_set_to_update = {
+            ob for ob in objects if isinstance(ob, DataSource)
+        }
+
+        for fv in views_to_update:
+            data_sources_set_to_update.add(fv.batch_source)
+            if fv.stream_source:
+                data_sources_set_to_update.add(fv.stream_source)
+
+        for rfv in request_views_to_update:
+            data_sources_set_to_update.add(rfv.request_data_source)
+
+        for odfv in odfvs_to_update:
+            for v in odfv.input_request_data_sources.values():
+                data_sources_set_to_update.add(v)
+
+        data_sources_to_update = list(data_sources_set_to_update)
 
         # Validate all feature views and make inferences.
         self._validate_all_feature_views(


Test patch: