Repository: feast-dev/feast

Issue:
Feast push (Redshift/DynamoDb) not work with PushMode.ONLINE_AND_OFFLINE when more than 500 columns
## Expected Behavior 
Currently, we have a push source with Redshift Offline Store and DynamoDb  Online Store. 
We built our view with more than 500 columns. Around 750 columns. 

We expected to ingest data in dynamo and redshift when we run 
`fs.push("push_source", df, to=PushMode.ONLINE_AND_OFFLINE)`

## Current Behavior

Push command raise an issue like` [ERROR] ValueError: The input dataframe has columns ..`
This issue come from `get_table_column_names_and_types `method in `write_to_offline_store` method.
In the method, we check if `if set(input_columns) != set(source_columns)` and raise the below issue if there are diff. 

In case with more than 500 columns we get a diff because source_columns come from `get_table_column_names_and_types` method result where the result is define by MaxResults parameters. 

## Steps to reproduce
```
entity= Entity(
    name="entity",
    join_keys=["entity_id"],
    value_type=ValueType.INT64,
)

push_source = PushSource(
    name="push_source",
    batch_source=RedshiftSource(
        table="fs_push_view",
        timestamp_field="datecreation",
        created_timestamp_column="created_at"),
)

besoin_embedding_push_view = FeatureView(
    name="push_view",
    entities=[entity],
    schema=[Field(name=f"field_{dim}", dtype=types.Float64) for dim in range(768)],
    source=push_source 
)

fs.push("push_source", df, to=PushMode.ONLINE_AND_OFFLINE)

```

### Specifications

- Version: 0.25.0
- Platform: AWS
- Subsystem:

## Possible Solution

In my mind, we have two solutions:

- Set higher MaxResults in describe_table method
- Use NextToken to iterate through results




Discussion:
Hi @beubeu13220 , I think either of the two solutions are good options. I think I'd prefer the `NextToken` approach simply because it's probably the most stable one.

Would you like to make a PR to add this functionality? We'd be happy to review!

Hi @achals,

Yes, I'll do that as soon as I have time.
For the moment, we use custom `write_to_offline_redshift function`. 

Resolving patch:
diff --git a/sdk/python/feast/infra/offline_stores/redshift_source.py b/sdk/python/feast/infra/offline_stores/redshift_source.py
--- a/sdk/python/feast/infra/offline_stores/redshift_source.py
+++ b/sdk/python/feast/infra/offline_stores/redshift_source.py
@@ -206,7 +206,8 @@ def get_table_column_names_and_types(
         client = aws_utils.get_redshift_data_client(config.offline_store.region)
         if self.table:
             try:
-                table = client.describe_table(
+                paginator = client.get_paginator("describe_table")
+                response_iterator = paginator.paginate(
                     ClusterIdentifier=config.offline_store.cluster_id,
                     Database=(
                         self.database
@@ -217,6 +218,7 @@ def get_table_column_names_and_types(
                     Table=self.table,
                     Schema=self.schema,
                 )
+                table = response_iterator.build_full_result()
             except ClientError as e:
                 if e.response["Error"]["Code"] == "ValidationException":
                     raise RedshiftCredentialsError() from e


Test patch: