internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: aws/aws-sdk-pandas

Issue:
Add data_source param to athena.repair_table
Hi, I can't seem to find where to specify data_source to repair_table method in athena api. Can you please add this param since our glue catalog is in a different account? Thank you. Pasting the current repair_table api below:

Parameters
table (str) – Table name.
database (str, optional) – AWS Glue/Athena database name.
s3_output (str, optional) – AWS S3 path.
workgroup (str, optional) – Athena workgroup.
encryption (str, optional) – None, ‘SSE_S3’, ‘SSE_KMS’, ‘CSE_KMS’.
kms_key (str, optional) – For SSE-KMS and CSE-KMS , this is the KMS key ARN or ID.
boto3_session (boto3.Session(), optional) – Boto3 Session. The default boto3 session will be used if boto3_session receive None.

Returns
Query final state (‘SUCCEEDED’, ‘FAILED’, ‘CANCELLED’).

Return type
str


Discussion:

Resolving patch:
diff --git a/awswrangler/athena/_utils.py b/awswrangler/athena/_utils.py
--- a/awswrangler/athena/_utils.py
+++ b/awswrangler/athena/_utils.py
@@ -505,6 +505,7 @@ def start_query_execution(
 def repair_table(
     table: str,
     database: Optional[str] = None,
+    data_source: Optional[str] = None,
     s3_output: Optional[str] = None,
     workgroup: Optional[str] = None,
     encryption: Optional[str] = None,
@@ -530,6 +531,8 @@ def repair_table(
         Table name.
     database : str, optional
         AWS Glue/Athena database name.
+    data_source : Optional[str], optional
+        Data Source / Catalog name. If None, 'AwsDataCatalog' is used.
     s3_output : str, optional
         AWS S3 path.
     workgroup : str, optional
@@ -559,6 +562,7 @@ def repair_table(
     query_id = start_query_execution(
         sql=query,
         database=database,
+        data_source=data_source,
         s3_output=s3_output,
         workgroup=workgroup,
         encryption=encryption,


Test patch:
Repository: aws/aws-sdk-pandas

Issue:
wr.config.s3_endpoint_url is not loaded
### Describe the bug

Env variable `WR_S3_ENDPOINT_URL` is not loaded into the `s3_endpoint_url` variable in `wr.config`. 

Setting it manually by `wr.config.s3_endpoint_url = os.environ['WR_S3_ENDPOINT_URL']` solves the problem.


### How to Reproduce

```
# In[1]:
import pandas as pd
import awswrangler as wr

# In[2]:
wr.__version__

# Out[2]:
'2.20.0'

# In[3]:
wr.config.s3_endpoint_url

# Out[3]:


# In[4]:
wr.s3.does_object_exist("path_to_file")

# Out[4]:
---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
...
ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
```


### Expected behavior

```
# In[1]:
import pandas as pd
import awswrangler as wr

# In[2]:
wr.__version__

# Out[2]:
'2.20.0'

# In[3]:
wr.config.s3_endpoint_url

# Out[3]:
# returns endpoint url

# In[4]:
wr.s3.does_object_exist("path_to_file")

# Out[4]:
True
```


### Your project

_No response_

### Screenshots

_No response_

### OS

Linux

### Python version

3.9.10

### AWS SDK for pandas version

2.20.0

### Additional context

_No response_


Discussion:
@LeonLuttenberger I think this is a regression from https://github.com/aws/aws-sdk-pandas/pull/2075

Specifically, when `loaded=True`, [L145](https://github.com/aws/aws-sdk-pandas/blob/1003d9720b83b4631eb03b33b23e22e3d2e00814/awswrangler/_config.py#L145) is preventing the environment variable from being loaded via L150.

I have tested with version 2.19.0 of the library and the endpoint is read from the env variable but not in 2.20.0

Resolving patch:
diff --git a/awswrangler/_config.py b/awswrangler/_config.py
--- a/awswrangler/_config.py
+++ b/awswrangler/_config.py
@@ -144,12 +144,12 @@ def to_pandas(self) -> pd.DataFrame:
     def _load_config(self, name: str) -> bool:
         if _CONFIG_ARGS[name].loaded:
             self._set_config_value(key=name, value=_CONFIG_ARGS[name].default)
-            return True
 
         env_var: Optional[str] = os.getenv(f"WR_{name.upper()}")
         if env_var is not None:
             self._set_config_value(key=name, value=env_var)
             return True
+
         return False
 
     def _set_config_value(self, key: str, value: Any) -> None:


Test patch:
diff --git a/tests/test_config.py b/tests/test_config.py
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -258,3 +258,13 @@ def test_athena_wait_delay_config_override(wr: ModuleType, glue_database: str) -
         mock_wait_query.assert_called_once()
 
         assert mock_wait_query.call_args[1]["athena_query_wait_polling_delay"] == polling_delay_argument
+
+
+@pytest.mark.parametrize("suppress_warnings", [False, True])
+def test_load_from_env_variable(wr: ModuleType, suppress_warnings: bool) -> None:
+    env_variable_value = "1" if suppress_warnings else ""
+
+    with patch.dict(os.environ, {"WR_SUPPRESS_WARNINGS": env_variable_value}):
+        wr.config.reset()
+
+        assert wr.config.suppress_warnings == suppress_warnings
Repository: aws/aws-sdk-pandas

Issue:
Provide option to skip commit in postgresql.to_sql to allow usage of transactions
**Is your idea related to a problem? Please describe.**
Currently, if there are no exceptions when calling awswrangler.postgresql.to_sql then commit is automatically done:

https://github.com/aws/aws-sdk-pandas/blob/b13fcd8d169feb3219b4b4fff025dc6089cfe03b/awswrangler/postgresql.py#L359

It makes it impossible to use awswrangler.postgresql.to_sql as a part of a transaction which e.g. writes multiple tables, because if one of the transaction steps fails, you cannot rollback all the changes, just the last one.

**Describe the solution you'd like**
To add extra parameter for awswrangler.postgresql.to_sql as it was done here, but for redshift:
https://github.com/aws/aws-sdk-pandas/issues/705




Discussion:

Resolving patch:
diff --git a/awswrangler/postgresql.py b/awswrangler/postgresql.py
--- a/awswrangler/postgresql.py
+++ b/awswrangler/postgresql.py
@@ -502,6 +502,7 @@ def to_sql(
     chunksize: int = 200,
     upsert_conflict_columns: list[str] | None = None,
     insert_conflict_columns: list[str] | None = None,
+    commit_transaction: bool = True,
 ) -> None:
     """Write records stored in a DataFrame into PostgreSQL.
 
@@ -542,6 +543,8 @@ def to_sql(
     insert_conflict_columns: List[str], optional
         This parameter is only supported if `mode` is set top `append`. In this case conflicts for the given columns are
         checked for evaluating the insert 'ON CONFLICT DO NOTHING'.
+    commit_transaction: bool
+        Whether to commit the transaction. True by default.
 
     Returns
     -------
@@ -606,7 +609,8 @@ def to_sql(
                 sql: str = f"INSERT INTO {_identifier(schema)}.{_identifier(table)} {insertion_columns} VALUES {placeholders}{upsert_str}"
                 _logger.debug("sql: %s", sql)
                 cursor.executemany(sql, (parameters,))
-            con.commit()
+            if commit_transaction:
+                con.commit()
     except Exception as ex:
         con.rollback()
         _logger.error(ex)


Test patch:
Repository: aws/aws-sdk-pandas

Issue:
Incorrect error message or implementation for datetime formatting?
### Describe the bug

when starting a SQL query with timezone aware date times, you get an error message like:

`Supports only timezone aware datatype, got 2024-04-29 13:30:34.573187+00:00`

The SDK code involved is

```python
     39 if value.tzinfo is not None:
     40     print(type(value))
---> 41     raise TypeError(f"Supports only timezone aware datatype, got {value}. {type(value)}")
     43 return f"TIMESTAMP '{value.isoformat(sep=' ', timespec='milliseconds')}'"
```
Where it checks for the existence of timezone info. And enter the if clause, if there is one.

However the error message say it support _only_ timezone aware dates.

### How to Reproduce

``` python
import awswrangler as wr
from datetime import timezone, datetime

wr.athena.start_query_execution(
  database='ddd',
  sql=f"""DELETE FROM ddd.t
                WHERE event_date >= :start 
         """,
    params={
        "start": datetime.now(timezone.utc),
    })
``````


### Expected behavior

To either have an error message like: 

`Supports only timezone naive datatype, got 2024-04-29 13:30:34.573187+00:00`

or: 

to correctly format timetamps with timezone info.

### Your project

Private

### Screenshots

_No response_

### OS

Mac

### Python version

3.11

### AWS SDK for pandas version

seen on main branch and ('3.4.2')

### Additional context

_No response_


Discussion:

Resolving patch:
diff --git a/awswrangler/_sql_formatter.py b/awswrangler/_sql_formatter.py
--- a/awswrangler/_sql_formatter.py
+++ b/awswrangler/_sql_formatter.py
@@ -40,7 +40,7 @@ def format_decimal(self, value: decimal.Decimal) -> str:
 
     def format_timestamp(self, value: datetime.datetime) -> str:
         if value.tzinfo is not None:
-            raise TypeError(f"Supports only timezone aware datatype, got {value}.")
+            raise TypeError(f"Supports only timezone naive datatype, got {value}.")
 
         return f"TIMESTAMP '{value.isoformat(sep=' ', timespec='milliseconds')}'"
 
@@ -132,7 +132,7 @@ def format_decimal(self, value: decimal.Decimal) -> str:
 
     def format_timestamp(self, value: datetime.datetime) -> str:
         if value.tzinfo is not None:
-            raise TypeError(f"Supports only timezone aware datatype, got {value}.")
+            raise TypeError(f"Supports only timezone naive datatype, got {value}.")
 
         return f"'{value.isoformat()}'"
 


Test patch:
Repository: borgbackup/borg

Issue:
Progress flag doesn't handle unicode gracefully.
The progress flag sometimes produces corrupted output when ellipsizing with certain unicode characters, presumably because they can have non-standard width, so the next printed progress report doesn't fully erase them.  So for example, a file with kanji in the name will be backed up, and then the following file will look like this:

`3.60 TB O   3.58 TB C   3.58 TB D home/$USERNAME/data3/A...gren's Syndrome.flacd 漢 字 漢 字 .flac`

The extra characters stick around until another kanji file is written, or until the backup is completed.  



Discussion:

Resolving patch:
diff --git a/borg/cache.py b/borg/cache.py
--- a/borg/cache.py
+++ b/borg/cache.py
@@ -93,7 +93,7 @@ def create(self):
         with open(os.path.join(self.path, 'config'), 'w') as fd:
             config.write(fd)
         ChunkIndex().write(os.path.join(self.path, 'chunks').encode('utf-8'))
-        with open(os.path.join(self.path, 'files'), 'w') as fd:
+        with open(os.path.join(self.path, 'files'), 'wb') as fd:
             pass  # empty file
 
     def destroy(self):
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -400,9 +400,9 @@ def test_corrupted_repository(self):
         self.cmd('extract', '--dry-run', self.repository_location + '::test')
         self.cmd('check', self.repository_location)
         name = sorted(os.listdir(os.path.join(self.tmpdir, 'repository', 'data', '0')), reverse=True)[0]
-        with open(os.path.join(self.tmpdir, 'repository', 'data', '0', name), 'r+') as fd:
+        with open(os.path.join(self.tmpdir, 'repository', 'data', '0', name), 'r+b') as fd:
             fd.seek(100)
-            fd.write('XXXX')
+            fd.write(b'XXXX')
         self.cmd('check', self.repository_location, exit_code=1)
 
     def test_readonly_repository(self):


Test patch:
Repository: borgbackup/borg

Issue:
borg delete repo asks again if you do not answer "YES"
workaround: Ctrl-C



Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -296,10 +296,11 @@ def do_delete(self, args):
             print("You requested to completely DELETE the repository *including* all archives it contains:")
             for archive_info in manifest.list_archive_infos(sort_by='ts'):
                 print(format_archive(archive_info))
-            while not os.environ.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'):
+            if not os.environ.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'):
                 print("""Type "YES" if you understand this and want to continue.\n""")
-                if input('Do you want to continue? ') == 'YES':
-                    break
+                if input('Do you want to continue? ') != 'YES':
+                    self.exit_code = 1
+                    return self.exit_code
             repository.destroy()
             cache.destroy()
             print("Repository and corresponding cache were deleted.")


Test patch:
Repository: borgbackup/borg

Issue:
crash when reading corrupt lock.roster file
maybe it's just my fiddling around in #231, but when i interrupt `borg convert`, it leaves a `lock.roster` file lying around. then when i try to run it again, i get an exception. i can't reproduce now, unfortunately, but there was an empty `lock.roster` file lying around. doing `rm lock.roster` solve the problem.

maybe some weird race condition?



Discussion:
oh, empty lock roster shouldn't happen, it should have json content.

yeah, somehow i can't reproduce... odd.

Guess the code dealing with lock.roster needs checking / fixing so it can deal with that without crashing.

Just to clarify: presence of a lock.roster file is not a bug. other than the exclusive lock directory, the roster file is not removed. but it shouldn't crash of course if the roster file's content is corrupt.


Resolving patch:
diff --git a/borg/locking.py b/borg/locking.py
--- a/borg/locking.py
+++ b/borg/locking.py
@@ -169,6 +169,9 @@ def load(self):
             if err.errno != errno.ENOENT:
                 raise
             data = {}
+        except ValueError:
+            # corrupt/empty roster file?
+            data = {}
         return data
 
     def save(self, data):


Test patch:
Repository: borgbackup/borg

Issue:
"patterns" missing from documentation
the `patterns` help topic is missing from the online documentation... it's not generated by `setup.py` either, so a little trickier than #279.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -150,19 +150,25 @@ def run(self):
         if not os.path.exists('docs/usage'):
             os.mkdir('docs/usage')
         for command, parser in choices.items():
-            if command is 'help':
-                continue
+            print('generating help for %s' % command)
             with open('docs/usage/%s.rst.inc' % command, 'w') as doc:
-                print('generating help for %s' % command)
-                params = {"command": command,
-                          "underline": '-' * len('borg ' + command)}
-                doc.write(".. _borg_{command}:\n\n".format(**params))
-                doc.write("borg {command}\n{underline}\n::\n\n".format(**params))
-                epilog = parser.epilog
-                parser.epilog = None
-                doc.write(re.sub("^", "    ", parser.format_help(), flags=re.M))
-                doc.write("\nDescription\n~~~~~~~~~~~\n")
-                doc.write(epilog)
+                if command == 'help':
+                    for topic in Archiver.helptext:
+                        params = {"topic": topic,
+                                  "underline": '~' * len('borg help ' + topic)}
+                        doc.write(".. _borg_{topic}:\n\n".format(**params))
+                        doc.write("borg help {topic}\n{underline}\n::\n\n".format(**params))
+                        doc.write(Archiver.helptext[topic])
+                else:
+                    params = {"command": command,
+                              "underline": '-' * len('borg ' + command)}
+                    doc.write(".. _borg_{command}:\n\n".format(**params))
+                    doc.write("borg {command}\n{underline}\n::\n\n".format(**params))
+                    epilog = parser.epilog
+                    parser.epilog = None
+                    doc.write(re.sub("^", "    ", parser.format_help(), flags=re.M))
+                    doc.write("\nDescription\n~~~~~~~~~~~\n")
+                    doc.write(epilog)
         # return to regular Cython configuration, if we changed it
         if os.environ.get('BORG_CYTHON_DISABLE') == self.__class__.__name__:
             del os.environ['BORG_CYTHON_DISABLE']


Test patch:
Repository: borgbackup/borg

Issue:
locking issue with FUSE mount
Reproduce like this:

```
borg mount repo mnt
fusermount -u mnt
borg delete repo
# will hang here waiting for exclusive lock
```



Discussion:
Idea: PID of locking process is different from the unlocking process' PID due to the fork. That likely creates a mismatch there and might make the unlock fail (or think that there is not "his" lock).


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -348,18 +348,21 @@ def do_mount(self, args):
             return self.exit_code
 
         repository = self.open_repository(args.src)
-        manifest, key = Manifest.load(repository)
-        if args.src.archive:
-            archive = Archive(repository, key, manifest, args.src.archive)
-        else:
-            archive = None
-        operations = FuseOperations(key, repository, manifest, archive)
-        self.print_verbose("Mounting filesystem")
         try:
-            operations.mount(args.mountpoint, args.options, args.foreground)
-        except RuntimeError:
-            # Relevant error message already printed to stderr by fuse
-            self.exit_code = EXIT_ERROR
+            manifest, key = Manifest.load(repository)
+            if args.src.archive:
+                archive = Archive(repository, key, manifest, args.src.archive)
+            else:
+                archive = None
+            operations = FuseOperations(key, repository, manifest, archive)
+            self.print_verbose("Mounting filesystem")
+            try:
+                operations.mount(args.mountpoint, args.options, args.foreground)
+            except RuntimeError:
+                # Relevant error message already printed to stderr by fuse
+                self.exit_code = EXIT_ERROR
+        finally:
+            repository.close()
         return self.exit_code
 
     def do_list(self, args):
diff --git a/borg/locking.py b/borg/locking.py
--- a/borg/locking.py
+++ b/borg/locking.py
@@ -2,7 +2,6 @@
 import json
 import os
 import socket
-import threading
 import time
 
 from borg.helpers import Error
@@ -10,13 +9,17 @@
 ADD, REMOVE = 'add', 'remove'
 SHARED, EXCLUSIVE = 'shared', 'exclusive'
 
+# only determine the PID and hostname once.
+# for FUSE mounts, we fork a child process that needs to release
+# the lock made by the parent, so it needs to use the same PID for that.
+_pid = os.getpid()
+_hostname = socket.gethostname()
+
 
 def get_id():
     """Get identification tuple for 'us'"""
-    hostname = socket.gethostname()
-    pid = os.getpid()
-    tid = threading.current_thread().ident & 0xffffffff
-    return hostname, pid, tid
+    thread_id = 0
+    return _hostname, _pid, thread_id
 
 
 class TimeoutTimer:


Test patch:
Repository: borgbackup/borg

Issue:
borg list --prefix
to just list the archives starting with the given prefix.



Discussion:

Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -414,6 +414,8 @@ def do_list(self, args):
                         remove_surrogates(item[b'path']), extra))
         else:
             for archive_info in manifest.list_archive_infos(sort_by='ts'):
+                if args.prefix and not archive_info.name.startswith(args.prefix):
+                    continue
                 print(format_archive(archive_info))
         return self.exit_code
 
@@ -835,9 +837,12 @@ def build_parser(self, args=None, prog=None):
         subparser.add_argument('--short', dest='short',
                                action='store_true', default=False,
                                help='only print file/directory names, nothing else')
+        subparser.add_argument('-p', '--prefix', dest='prefix', type=str,
+                               help='only consider archive names starting with this prefix')
         subparser.add_argument('src', metavar='REPOSITORY_OR_ARCHIVE', nargs='?', default='',
                                type=location_validator(),
                                help='repository/archive to list contents of')
+
         mount_epilog = textwrap.dedent("""
         This command mounts an archive as a FUSE filesystem. This can be useful for
         browsing an archive or restoring individual files. Unless the ``--foreground``
diff --git a/borg/testsuite/archiver.py b/borg/testsuite/archiver.py
--- a/borg/testsuite/archiver.py
+++ b/borg/testsuite/archiver.py
@@ -675,6 +675,16 @@ def test_prune_repository_prefix(self):
         self.assert_in('bar-2015-08-12-10:00', output)
         self.assert_in('bar-2015-08-12-20:00', output)
 
+    def test_list_prefix(self):
+        self.cmd('init', self.repository_location)
+        self.cmd('create', self.repository_location + '::test-1', src_dir)
+        self.cmd('create', self.repository_location + '::something-else-than-test-1', src_dir)
+        self.cmd('create', self.repository_location + '::test-2', src_dir)
+        output = self.cmd('list', '--prefix=test-', self.repository_location)
+        self.assert_in('test-1', output)
+        self.assert_in('test-2', output)
+        self.assert_not_in('something-else', output)
+
     def test_usage(self):
         if self.FORK_DEFAULT:
             self.cmd(exit_code=0)


Test patch:
Repository: borgbackup/borg

Issue:
create -v documentation
I am unsure what the prefix mean:

M - modified
U - unchanged
d - directory
A -?



Discussion:
Yes, that is still missing from the docs. I just used ad-hoc letters. Maybe we could review the letters and use same as `ls -l` outputs?

See return / status values there:
https://github.com/borgbackup/borg/blob/master/borg/archive.py#L457

See comment there:
https://github.com/borgbackup/borg/blob/master/borg/archiver.py#L248

mmh. 
i repeated to add the same unchanged/untouched directory like 
create ..repo::a then create... samerepo:b
but the last file always gets the "A"
...so there must be a bug...

Please create a separate issue for that (try to create a minimal, reproducable case for that and don't forget to give the borgbackup version).


Resolving patch:
diff --git a/borg/archiver.py b/borg/archiver.py
--- a/borg/archiver.py
+++ b/borg/archiver.py
@@ -245,13 +245,6 @@ def _process(self, archive, cache, excludes, exclude_caches, exclude_if_present,
             self.print_warning('Unknown file type: %s', path)
             return
         # Status output
-        # A lowercase character means a file type other than a regular file,
-        # borg usually just stores them. E.g. (d)irectory.
-        # Hardlinks to already seen content are indicated by (h).
-        # A uppercase character means a regular file that was (A)dded,
-        # (M)odified or was (U)nchanged.
-        # Note: A/M/U is relative to the "files" cache, not to the repo.
-        # This would be an issue if the files cache is not used.
         if status is None:
             if not dry_run:
                 status = '?'  # need to add a status code somewhere


Test patch: