internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: facebookresearch/habitat-lab

Issue:
DD-PPO does not all reduce gradients
## 🐛 Bug

DD-PPO does not all reduce gradients during the backward call, because `reducer.prepare_for_backward` is not being called during training process.

The problem is in this line: https://github.com/facebookresearch/habitat-api/blob/v0.1.4/habitat_baselines/rl/ddppo/algo/ddppo.py#L96

```
class DecentralizedDistributedMixin:

    ...
    def before_backward(self, loss):
         # ...
         self.reducer.prepare_for_backward(..)


# Mixin goes second that way the PPO __init__ will still be called
class DDPPO(PPO, DecentralizedDistributedMixin):  
    # Here PPO and Mixin both have "before_backward" method, 
    # DDPPO will call PPO's not the Mixin's.
    pass
```

And here is a quick fix:
```
class DecentralizedDistributedMixin:
    ...


# Mixin goes second that way the PPO __init__ will still be called
class DDPPO(PPO, DecentralizedDistributedMixin):  

      # Move before_backward to DDPPO
      def before_backward(self, loss):
         # ...
         self.reducer.prepare_for_backward(..)
```



Discussion:

Resolving patch:
diff --git a/habitat_baselines/rl/ddppo/algo/ddppo.py b/habitat_baselines/rl/ddppo/algo/ddppo.py
--- a/habitat_baselines/rl/ddppo/algo/ddppo.py
+++ b/habitat_baselines/rl/ddppo/algo/ddppo.py
@@ -92,6 +92,5 @@ def before_backward(self, loss):
             self.reducer.prepare_for_backward([])
 
 
-# Mixin goes second that way the PPO __init__ will still be called
-class DDPPO(PPO, DecentralizedDistributedMixin):
+class DDPPO(DecentralizedDistributedMixin, PPO):
     pass


Test patch:
Repository: facebookresearch/habitat-lab

Issue:
Have Default SensorTypes have a YAML Customizable UUIDs
## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->
* Currently, using multiple visual sensors is difficult since each sensor has a UUID that is baked into the default class. It should be able to be overriden by the configs.
## Motivation
It should be able to be overriden for the configs for the default RGBs, Depth Sensors, Semantic etc at the very least. That way, one can define multiple RGB, Depth, or other sensors without having to subclass each sensor.
<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

## Pitch

<!-- A clear and concise description of what you want to happen. -->
Being able to define multiple sensors from pure YAML without needing to subclass the model.
## Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->



Discussion:

Resolving patch:
diff --git a/habitat/core/simulator.py b/habitat/core/simulator.py
--- a/habitat/core/simulator.py
+++ b/habitat/core/simulator.py
@@ -63,7 +63,11 @@ class Sensor:
 
     def __init__(self, *args: Any, **kwargs: Any) -> None:
         self.config = kwargs["config"] if "config" in kwargs else None
-        self.uuid = self._get_uuid(*args, **kwargs)
+        if hasattr(self.config, "UUID"):
+            # We allow any sensor config to override the UUID
+            self.uuid = self.config.UUID
+        else:
+            self.uuid = self._get_uuid(*args, **kwargs)
         self.sensor_type = self._get_sensor_type(*args, **kwargs)
         self.observation_space = self._get_observation_space(*args, **kwargs)
 


Test patch:
Repository: facebookresearch/habitat-lab

Issue:
Unique seed not set for each simulator
Habitat-Lab: v0.1.7 & master

## 🐛 Bug

Each environment should get a unique seed, but it looks like each process in the process group is provided the same seed in this code: https://github.com/facebookresearch/habitat-lab/blob/a7eb39fb3b420f9e6f528796f97457715bd79a09/habitat_baselines/rl/ppo/ppo_trainer.py#L217
```
# Multiply by the number of simulators to make sure they also get unique seeds
self.config.TASK_CONFIG.SEED += (
    torch.distributed.get_world_size()
    * self.config.NUM_ENVIRONMENTS
)
```

## Expected behavior
As far as I understand, `torch.distributed.get_world_size()` is a fixed number (the number of processes in the process group). Instead, shouldn't `local_rank` be used to modify the seed?
```
# Multiply by the number of simulators to make sure they also get unique seeds
self.config.TASK_CONFIG.SEED += (
    local_rank
    * self.config.NUM_ENVIRONMENTS
)
```


Discussion:
CC: @mathfac 

Resolving patch:
diff --git a/habitat_baselines/rl/ppo/ppo_trainer.py b/habitat_baselines/rl/ppo/ppo_trainer.py
--- a/habitat_baselines/rl/ppo/ppo_trainer.py
+++ b/habitat_baselines/rl/ppo/ppo_trainer.py
@@ -216,8 +216,7 @@ def _init_train(self):
             self.config.SIMULATOR_GPU_ID = local_rank
             # Multiply by the number of simulators to make sure they also get unique seeds
             self.config.TASK_CONFIG.SEED += (
-                torch.distributed.get_world_size()
-                * self.config.NUM_ENVIRONMENTS
+                torch.distributed.get_rank() * self.config.NUM_ENVIRONMENTS
             )
             self.config.freeze()
 


Test patch:
Repository: facebookresearch/habitat-lab

Issue:
[BUG] FOV Is not properly set
## Habitat-Lab and Habitat-Sim versions
Master

## 🐛 Bug
The HFOV config value is ignored.  During the recent heirarchical refactor, the FOV value was removed from the SensorSpec. Now it is not used at all by HabitatSimulator since it is not set by the SensorSpec or at runtime.
## Steps to Reproduce

Steps to reproduce the behavior:

1. Change the HFOV
2. Compared the output
3. Realize it's not read in at all.

Please note that without a minimal working example to reproduce the bug, we may not be able to help you.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior
The config value should change the FOV of the habitat.
<!-- A clear and concise description of what you expected to happen. -->

## Additional context
* This was broken and not caught by tests due to use not comparing FOV values for sensor ground truth. We should consider adding this into our sensor suite.
<!-- Add any other context about the problem here. -->



Discussion:
Bug was created by PR #598 

Resolving patch:
diff --git a/habitat/sims/habitat_simulator/habitat_simulator.py b/habitat/sims/habitat_simulator/habitat_simulator.py
--- a/habitat/sims/habitat_simulator/habitat_simulator.py
+++ b/habitat/sims/habitat_simulator/habitat_simulator.py
@@ -304,7 +304,6 @@ def create_sim_config(
                 # or translated into the sensor config manually
                 ignore_keys={
                     "height",
-                    "hfov",
                     "max_depth",
                     "min_depth",
                     "normalize_depth",


Test patch:
Repository: facebookresearch/habitat-lab

Issue:
Should `VelocityControlActionConfig` be added to the ConfigStore
[This](https://github.com/facebookresearch/habitat-lab/blob/f64ca16901947947cb5a74ede34e5325f2b9f4fc/habitat-lab/habitat/config/default_structured_configs.py#L156) config option doesn't seem to be added to the config store, is this deliberate? I was trying to do something like the following:
```yaml
@package _global_
defaults:
  - /habitat/task/actions:
    - velocity_control
```
However, I had to add the following to the default configs to make this work (I didn't see any existing entries for this config option)
```python
cs.store(
    package="habitat.task.actions.velocity_control",
    group="habitat/task/actions",
    name="velocity_control",
    node=VelocityControlActionConfig,
)
```
Do let me know if this is not the intended way for this to be used. Thanks.

cc @jimmytyyang @KRobG


Discussion:
Thanks for reporting this. I think this is not intended

Resolving patch:
diff --git a/habitat-lab/habitat/config/default_structured_configs.py b/habitat-lab/habitat/config/default_structured_configs.py
--- a/habitat-lab/habitat/config/default_structured_configs.py
+++ b/habitat-lab/habitat/config/default_structured_configs.py
@@ -1314,6 +1314,12 @@ class HabitatConfig(HabitatBaseConfig):
     name="base_velocity",
     node=BaseVelocityActionConfig,
 )
+cs.store(
+    package="habitat.task.actions.velocity_control",
+    group="habitat/task/actions",
+    name="velocity_control",
+    node=VelocityControlActionConfig,
+)
 cs.store(
     package="habitat.task.actions.empty",
     group="habitat/task/actions",


Test patch:
Repository: astronomer/astro-sdk

Issue:
Unable to load_file using parquet
Version: Astro 0.2.0
Python: 3.8, 3.9

Astro is unable to run the task `load_file` with a parquet file.

It raises the following exception:
```
  Traceback (most recent call last):
    File "pyarrow/io.pxi", line 1511, in pyarrow.lib.get_native_file
    File "/home/tati/.virtualenvs/astro-py38/lib/python3.8/site-packages/pyarrow/util.py", line 99, in _stringify_path
      raise TypeError("not a path-like object")
  TypeError: not a path-like object
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "pyarrow/io.pxi", line 1517, in pyarrow.lib.get_native_file
    File "pyarrow/io.pxi", line 729, in pyarrow.lib.PythonFile.__cinit__
  TypeError: binary file expected, got text file
  
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
```


Discussion:

Resolving patch:
diff --git a/src/astro/sql/operators/agnostic_load_file.py b/src/astro/sql/operators/agnostic_load_file.py
--- a/src/astro/sql/operators/agnostic_load_file.py
+++ b/src/astro/sql/operators/agnostic_load_file.py
@@ -119,8 +119,11 @@ def _load_dataframe(self, path):
             "json": pd.read_json,
             "ndjson": pd.read_json,
         }
+        mode = {"parquet": "rb"}
         deserialiser_params = {"ndjson": {"lines": True}}
-        with open(path, transport_params=transport_params) as stream:
+        with open(
+            path, mode=mode.get(file_type, "r"), transport_params=transport_params
+        ) as stream:
             return deserialiser[file_type](
                 stream, **deserialiser_params.get(file_type, {})
             )


Test patch:
diff --git a/tests/data/sample.parquet b/tests/data/sample.parquet
new file mode 100644
Binary files /dev/null and b/tests/data/sample.parquet differ
diff --git a/tests/operators/test_agnostic_load_file.py b/tests/operators/test_agnostic_load_file.py
--- a/tests/operators/test_agnostic_load_file.py
+++ b/tests/operators/test_agnostic_load_file.py
@@ -470,7 +470,7 @@ def test_aws_decode():
 
 
 @pytest.mark.parametrize("sql_server", ["snowflake", "postgres"], indirect=True)
-@pytest.mark.parametrize("file_type", ["ndjson", "json", "csv"])
+@pytest.mark.parametrize("file_type", ["parquet", "ndjson", "json", "csv"])
 def test_load_file(sample_dag, sql_server, file_type):
     sql_name, sql_hook = sql_server
Repository: astronomer/astro-sdk

Issue:
Add SQLite example
* Create an example in `example_dags` illustrating the usage of SQLite
* This example could use one of our checks
* Update `tests/test_example_dags.py` to run it


Discussion:
Ready for review: https://github.com/astro-projects/astro/pull/157

Resolving patch:
diff --git a/example_dags/example_sqlite_load_transform.py b/example_dags/example_sqlite_load_transform.py
--- a/example_dags/example_sqlite_load_transform.py
+++ b/example_dags/example_sqlite_load_transform.py
@@ -27,7 +27,7 @@ def top_five_animations(input_table: Table):
 ) as dag:
 
     imdb_movies = aql.load_file(
-        path="https://raw.githubusercontent.com/astro-projects/astro/readme/tests/data/imdb.csv",
+        path="https://raw.githubusercontent.com/astro-projects/astro/main/tests/data/imdb.csv",
         task_id="load_csv",
         output_table=Table(
             table_name="imdb_movies", database="sqlite", conn_id="sqlite_default"
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -34,8 +34,8 @@ def test(session: nox.Session) -> None:
     [
         ("postgres-only", {"include": ["postgres"], "exclude": ["amazon"]}),
         ("postgres-amazon", {"include": ["postgres", "amazon"]}),
-        ("snowflake-amazon", {"include": ["snowflake", "amazon"]})
-        # ("sqlite", {"include": ["sqlite"]}),
+        ("snowflake-amazon", {"include": ["snowflake", "amazon"]}),
+        ("sqlite", {"include": ["sqlite"]}),
     ],
 )
 def test_examples_by_dependency(session: nox.Session, extras):


Test patch:
diff --git a/tests/test_example_dags.py b/tests/test_example_dags.py
--- a/tests/test_example_dags.py
+++ b/tests/test_example_dags.py
@@ -31,7 +31,7 @@ def session():
         "example_postgres_render",
         "example_snowflake_partial_table_with_append",
         "example_snowflake_render",
-        # "example_sqlite_load_transform",
+        "example_sqlite_load_transform",
     ],
 )
 def test_example_dag(session, dag_id):
Repository: astronomer/astro-sdk

Issue:
Add `__repr__` to all our classes
**Reference**:
- https://stackoverflow.com/questions/1436703/what-is-the-difference-between-str-and-repr

Currently, without `__repr__`, if you create an instance of a Database class and print it out, it will give the memory address as shown below.
```python
>>> from astro.databases.snowflake import SnowflakeDatabase
>>> SnowflakeDatabase("d")
<astro.databases.snowflake.SnowflakeDatabase object at 0x7ff3c80afe80>
```

As part of this GitHub issue, we should evaluate all of custom classes (`FileType`, `FileLocation` etc) and add `__repr__` methods wherever they are missing.

Example:

https://github.com/astronomer/astro-sdk/blob/5a511d265c6aed10248a1e36fd986518eaec0875/src/astro/files/base.py#L87-L93


Discussion:
@kaxil should we add this to the operators as well? It seems like the only objects that users would interact with are Database, File, Table, and Metadata.

Resolving patch:
diff --git a/src/astro/databases/base.py b/src/astro/databases/base.py
--- a/src/astro/databases/base.py
+++ b/src/astro/databases/base.py
@@ -46,6 +46,9 @@ class BaseDatabase(ABC):
     def __init__(self, conn_id: str):
         self.conn_id = conn_id
 
+    def __repr__(self):
+        return f'{self.__class__.__name__}(conn_id="{self.conn_id})'
+
     @property
     def hook(self) -> BaseHook:
         """Return an instance of the database-specific Airflow hook."""
diff --git a/src/astro/files/types/base.py b/src/astro/files/types/base.py
--- a/src/astro/files/types/base.py
+++ b/src/astro/files/types/base.py
@@ -38,3 +38,6 @@ def name(self):
     def __str__(self):
         """String representation of type"""
         return self.name.value
+
+    def __repr__(self):
+        return f'{self.__class__.__name__}(path="{self.path}")'


Test patch:
Repository: astronomer/astro-sdk

Issue:
Row count not working for snowflake
**Describe the bug**
Currently row count fails for snowflake 

**Expected behavior**
Row count should work snowflake




Discussion:

Resolving patch:
diff --git a/python-sdk/src/astro/table.py b/python-sdk/src/astro/table.py
--- a/python-sdk/src/astro/table.py
+++ b/python-sdk/src/astro/table.py
@@ -5,7 +5,7 @@
 from typing import Any
 
 from attr import define, field, fields_dict
-from sqlalchemy import Column, MetaData, func, select
+from sqlalchemy import Column, MetaData
 
 from astro.airflow.datasets import Dataset
 from astro.databases import create_database
@@ -132,9 +132,10 @@ def row_count(self) -> Any:
         Return the row count of table.
         """
         db = create_database(self.conn_id)
-        tb = db.get_sqla_table(table=self)
-        query = select(func.count("*")).select_from(tb)
-        return db.run_sql(query).scalar()
+        result = db.run_sql(
+            f"select count(*) from {db.get_table_qualified_name(self)}"  # skipcq: BAN-B608
+        ).scalar()
+        return result
 
     def to_json(self):
         return {


Test patch:
diff --git a/python-sdk/tests/utils/test_table.py b/python-sdk/tests/utils/test_table.py
--- a/python-sdk/tests/utils/test_table.py
+++ b/python-sdk/tests/utils/test_table.py
@@ -2,10 +2,11 @@
 
 import pytest
 
+from astro.constants import Database
 from astro.files import File
 from astro.sql import LoadFileOperator
 from astro.sql.operators.transform import TransformOperator
-from astro.table import BaseTable, Metadata, Table
+from astro.table import BaseTable, Table
 from astro.utils.table import find_first_table
 
 
@@ -94,16 +95,39 @@ def test_find_first_table_with_xcom_arg(xcom_arg_resolve, kwargs, return_type):
 
 
 @pytest.mark.integration
-def test_row_count():
+@pytest.mark.parametrize(
+    "database_table_fixture",
+    [
+        {
+            "database": Database.SNOWFLAKE,
+        },
+        {
+            "database": Database.BIGQUERY,
+        },
+        {
+            "database": Database.POSTGRES,
+        },
+        {
+            "database": Database.SQLITE,
+        },
+        {
+            "database": Database.REDSHIFT,
+        },
+    ],
+    indirect=True,
+    ids=["snowflake", "bigquery", "postgresql", "sqlite", "redshift"],
+)
+def test_row_count(database_table_fixture):
     """
     Load file in bigquery and test the row count of bigquery table
     """
+    _, test_table = database_table_fixture
     imdb_table = LoadFileOperator(
         task_id="load_file",
         input_file=File(
             path="https://raw.githubusercontent.com/astronomer/astro-sdk/main/tests/data/imdb_v2.csv"
         ),
-        output_table=Table(conn_id="gcp_conn", metadata=Metadata(schema="astro")),
+        output_table=test_table,
     ).execute({})
 
-    assert imdb_table.row_count > 0
+    assert imdb_table.row_count == 117
Repository: astronomer/astro-sdk

Issue:
CI: `example_load_file.py` isnt running
**Describe the bug**
 The `python-sdk/example_dags/example_load_file.py` example DAG in not running in our CI tests

**Version**
* Astro: [e.g. 0.6.0]
* OS: [eg. Debian]

**To Reproduce**
Steps to reproduce the behavior:
1. Write the DAG  '...'
2. Create connection '....'
3. Run using  '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.


**Additional context**
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/python-sdk/example_dags/example_load_file.py b/python-sdk/example_dags/example_load_file.py
--- a/python-sdk/example_dags/example_load_file.py
+++ b/python-sdk/example_dags/example_load_file.py
@@ -142,7 +142,7 @@
     # [START load_file_example_12]
     aql.load_file(
         input_file=File(
-            "gs://astro-sdk/workspace/sample_pattern",
+            "gs://astro-sdk/workspace/sample_pattern.csv",
             conn_id="bigquery",
             filetype=FileType.CSV,
         ),
@@ -166,7 +166,7 @@
     # [START load_file_example_14]
     aql.load_file(
         input_file=File(
-            "gs://astro-sdk/workspace/sample_pattern",
+            "gs://astro-sdk/workspace/sample_pattern.csv",
             conn_id="bigquery",
             filetype=FileType.CSV,
         ),
@@ -200,7 +200,7 @@
     # [START load_file_example_17]
     aql.load_file(
         input_file=File(
-            "gs://astro-sdk/workspace/sample_pattern",
+            "gs://astro-sdk/workspace/sample_pattern.csv",
             conn_id="bigquery",
             filetype=FileType.CSV,
         ),


Test patch:
Repository: astronomer/astro-sdk

Issue:
dataframe operator is now appending to it's output table instead of replacing.
**Describe the bug**
 the dataframe operator is now appending to it's output table instead of replacing.
refer to https://astronomer.slack.com/archives/C02B8SPT93K/p1668643987896139

**Version**
* Astro: [e.g. 0.6.0]
* OS: [eg. Debian]

**To Reproduce**
Steps to reproduce the behavior:
1. Write the DAG  '...'
2. Create connection '....'
3. Run using  '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.


**Additional context**
Add any other context about the problem here.



Discussion:
This was caused after the upgrade from 1.1.1 (working) to 1.2.1 (causing dupes)

Resolving patch:
diff --git a/python-sdk/src/astro/databases/snowflake.py b/python-sdk/src/astro/databases/snowflake.py
--- a/python-sdk/src/astro/databases/snowflake.py
+++ b/python-sdk/src/astro/databases/snowflake.py
@@ -648,8 +648,10 @@ def load_pandas_dataframe_to_table(
         self._assert_not_empty_df(source_dataframe)
 
         auto_create_table = False
-        if if_exists == "replace" or not self.table_exists(target_table):
+        if not self.table_exists(target_table):
             auto_create_table = True
+        elif if_exists == "replace":
+            self.create_table(target_table, dataframe=source_dataframe)
 
         # We are changing the case of table name to ease out on the requirements to add quotes in raw queries.
         # ToDO - Currently, we cannot to append using load_file to a table name which is having name in lower case.


Test patch:
diff --git a/python-sdk/tests/sql/operators/test_dataframe.py b/python-sdk/tests/sql/operators/test_dataframe.py
--- a/python-sdk/tests/sql/operators/test_dataframe.py
+++ b/python-sdk/tests/sql/operators/test_dataframe.py
@@ -465,3 +465,33 @@ def get_empty_dataframe():
     with pytest.raises(ValueError) as exec_info:
         test_utils.run_dag(sample_dag)
     assert exec_info.value.args[0] == "Can't load empty dataframe"
+
+
+@pytest.mark.integration
+@pytest.mark.parametrize(
+    "conn_id",
+    [
+        "bigquery",
+        "postgres_conn",
+        "redshift_conn",
+        "snowflake_conn",
+        "sqlite_conn",
+    ],
+)
+def test_dataframe_replace_table_if_exist(sample_dag, conn_id):
+    @aql.dataframe
+    def get_empty_dataframe():
+        arr = {"col1": [1, 2]}
+        return pandas.DataFrame(data=arr)
+
+    output_tb = Table(
+        name="ci_df_replace_table",
+        conn_id=conn_id,
+    )
+    with sample_dag:
+        get_empty_dataframe(output_table=output_tb)
+    test_utils.run_dag(sample_dag)
+    assert output_tb.row_count == 2
+    # re-run dag to and make sure it is replacing table
+    test_utils.run_dag(sample_dag)
+    assert output_tb.row_count == 2
Repository: astronomer/astro-sdk

Issue:
Doc: Cross link to API reference page from Operators page
Currently there is no way to jump to Func/Operator definition from https://astro-sdk-python.readthedocs.io/en/stable/astro/sql/operators/get_value_list.html (and other operators listed on https://astro-sdk-python.readthedocs.io/en/stable/guides/operators.html) to their definition https://astro-sdk-python.readthedocs.io/en/stable/autoapi/index.html

We should cross-link them using https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#cross-referencing-syntax


Discussion:

Resolving patch:
diff --git a/python-sdk/src/astro/sql/__init__.py b/python-sdk/src/astro/sql/__init__.py
--- a/python-sdk/src/astro/sql/__init__.py
+++ b/python-sdk/src/astro/sql/__init__.py
@@ -24,6 +24,7 @@
     "drop_table",
     "ExportFileOperator",
     "export_file",
+    "get_value_list",
     "LoadFileOperator",
     "load_file",
     "MergeOperator",


Test patch: