Repository: feast-dev/feast

Issue:
Provide ability to set more options when outputting joined data to bq table.
**Is your feature request related to a problem? Please describe.**

We need to be able to set expiries for the tables, project.dataset.tableName, and also option for create vs replace.

**Describe the solution you'd like**

Ideally the user can optionally pass in a config object like QueryJobConfig. QueryJobConfig has many options such as `destination` which allows one to set the destination of table and name, but it doesn't seem like it can handle create/replace (it has create_disposition which offers some limited control) and expiry.

**Describe alternatives you've considered**

N/A. Willem and a few of us briefly discussed this and he played around with the idea of passing in and out sql queries. So one could modify the query and pass it back in. 


Discussion:
@mavysavydav the problem statement makes sense. `get_historical_features()` is a general (non-implementation specific) method, but the retrieval jobs are specific to the implementation. So roughly there are options like

1. 
```
job = get_historical_features(..., kwargs)
```
where those options get passed all the way to the offline store and the offline store then knows how to use them

2. 
```
job = get_historical_features(...)
job.set_config(QueryJobConfig)
job.to_bigquery()
```
3. 
```
job = get_historical_features(...)
job.to_bigquery(..., config=QueryJobConfig)
```
4. 
```
job = get_historical_features(...)
sql = job.to_sql()
job_config = bigquery.QueryJobConfig(default_dataset="bigquery-public-data.stackoverflow")
client.query(sql, job_config=job_config)
```

My preference is (3) or (4) right now, but don't feel super strongly.
:raising_hand: working on this now! Please assign me.
yea agreed 3 or 4 is good.  cody will look into 
solution 4 would work well with in-house wrappers. The wrapper could still present a clean api for users even tho it's changing the sql to add things like expiry. Though expiry seems like something feast would want to support natively. Tho maybe we'll wait for more needs to emerge b4 natively supporting it. 

Resolving patch:
diff --git a/sdk/python/feast/infra/offline_stores/bigquery.py b/sdk/python/feast/infra/offline_stores/bigquery.py
--- a/sdk/python/feast/infra/offline_stores/bigquery.py
+++ b/sdk/python/feast/infra/offline_stores/bigquery.py
@@ -224,31 +224,51 @@ def to_df(self):
         df = self.client.query(self.query).to_dataframe(create_bqstorage_client=True)
         return df
 
-    def to_bigquery(self, dry_run=False) -> Optional[str]:
+    def to_sql(self) -> str:
+        """
+        Returns the SQL query that will be executed in BigQuery to build the historical feature table.
+        """
+        return self.query
+
+    def to_bigquery(self, job_config: bigquery.QueryJobConfig = None) -> Optional[str]:
+        """
+        Triggers the execution of a historical feature retrieval query and exports the results to a BigQuery table.
+
+        Args:
+            job_config: An optional bigquery.QueryJobConfig to specify options like destination table, dry run, etc.
+
+        Returns:
+            Returns the destination table name or returns None if job_config.dry_run is True.
+        """
+
         @retry(wait=wait_fixed(10), stop=stop_after_delay(1800), reraise=True)
         def _block_until_done():
             return self.client.get_job(bq_job.job_id).state in ["PENDING", "RUNNING"]
 
-        today = date.today().strftime("%Y%m%d")
-        rand_id = str(uuid.uuid4())[:7]
-        dataset_project = self.config.offline_store.project_id or self.client.project
-        path = f"{dataset_project}.{self.config.offline_store.dataset}.historical_{today}_{rand_id}"
-        job_config = bigquery.QueryJobConfig(destination=path, dry_run=dry_run)
-        bq_job = self.client.query(self.query, job_config=job_config)
-
-        if dry_run:
-            print(
-                "This query will process {} bytes.".format(bq_job.total_bytes_processed)
+        if not job_config:
+            today = date.today().strftime("%Y%m%d")
+            rand_id = str(uuid.uuid4())[:7]
+            dataset_project = (
+                self.config.offline_store.project_id or self.client.project
             )
-            return None
+            path = f"{dataset_project}.{self.config.offline_store.dataset}.historical_{today}_{rand_id}"
+            job_config = bigquery.QueryJobConfig(destination=path)
+
+        bq_job = self.client.query(self.query, job_config=job_config)
 
         _block_until_done()
 
         if bq_job.exception():
             raise bq_job.exception()
 
-        print(f"Done writing to '{path}'.")
-        return path
+        if job_config.dry_run:
+            print(
+                "This query will process {} bytes.".format(bq_job.total_bytes_processed)
+            )
+            return None
+
+        print(f"Done writing to '{job_config.destination}'.")
+        return str(job_config.destination)
 
 
 @dataclass(frozen=True)


Test patch:
diff --git a/sdk/python/tests/test_historical_retrieval.py b/sdk/python/tests/test_historical_retrieval.py
--- a/sdk/python/tests/test_historical_retrieval.py
+++ b/sdk/python/tests/test_historical_retrieval.py
@@ -438,29 +438,13 @@ def test_historical_features_from_bigquery_sources(
             ],
         )
 
-        # Just a dry run, should not create table
-        bq_dry_run = job_from_sql.to_bigquery(dry_run=True)
-        assert bq_dry_run is None
-
-        bq_temp_table_path = job_from_sql.to_bigquery()
-        assert bq_temp_table_path.split(".")[0] == gcp_project
-
-        if provider_type == "gcp_custom_offline_config":
-            assert bq_temp_table_path.split(".")[1] == "foo"
-        else:
-            assert bq_temp_table_path.split(".")[1] == bigquery_dataset
-
-        # Check that this table actually exists
-        actual_bq_temp_table = bigquery.Client().get_table(bq_temp_table_path)
-        assert actual_bq_temp_table.table_id == bq_temp_table_path.split(".")[-1]
-
         start_time = datetime.utcnow()
         actual_df_from_sql_entities = job_from_sql.to_df()
         end_time = datetime.utcnow()
         with capsys.disabled():
             print(
                 str(
-                    f"\nTime to execute job_from_df.to_df() = '{(end_time - start_time)}'"
+                    f"\nTime to execute job_from_sql.to_df() = '{(end_time - start_time)}'"
                 )
             )