internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: bids-standard/pybids

Issue:
BIDSLayoutIndexer not passed to derivative layouts
The following will not respect the indexer preference for the derivative layouts

````
    indexer = BIDSLayoutIndexer(index_metadata=False)
    layout = BIDSLayout(preproc_dir, derivatives=preproc_dir, indexer=indexer)
```



Discussion:
I just noticed the same thing. When I checked the code, the issue is that the section that creates derivative BIDSLayouts was not updated to pass through the BIDSLayoutIndexer object, just the now deprecated "indexer_kwargs":

https://github.com/bids-standard/pybids/blob/fd7f0d92f62fbae0c6f77a03f38269b1782a5325/bids/layout/layout.py#L178-L187

A quick workaround I used was to pass in my indexer arguments to the BIDSLayout class the "old way," which lets these arguments pass through to the derivative layout:
``` python
layout = BIDSLayout(".../data_BIDS",
                    derivatives=".../data_BIDS/derivatives/fmriprep", reset_database=True,
                    validate=True, index_metadata=False) # Indexer args here
```

This was *crucial* for me using the fmriprep derivative - with 100 subjects, it was taking around 45mins to index. The key was that `index_metadata=False` was not applying, which takes much much longer. With this workaround I got my index made in about 5 mins.

Of course, you'll get a warning that you should be using the Index object, which you have to ignore here.


I believe the fix would be to pass the indexer object through to `add_derivatives()` so that it can be applied to the derivative dataset here:

https://github.com/bids-standard/pybids/blob/fd7f0d92f62fbae0c6f77a03f38269b1782a5325/bids/layout/layout.py#L537

May give this a shot myself if I can figure out how to contribute properly.

Resolving patch:
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -169,11 +169,12 @@ def __init__(self, root=None, validate=True, absolute_paths=True,
             self.connection_manager = ConnectionManager(
                 database_path, reset_database, config, init_args)
 
-            if indexer is None:
-                indexer = BIDSLayoutIndexer(
-                    validate=validate and not is_derivative, **indexer_kwargs
-                )
-            indexer(self)
+            # Do not overwrite indexer variable, so the same configuration is passed to
+            # add_derivatives() below
+            _indexer = indexer or BIDSLayoutIndexer(
+                validate=validate and not is_derivative, **indexer_kwargs
+            )
+            _indexer(self)
 
         # Add derivatives if any are found
         if derivatives:
@@ -184,7 +185,7 @@ def __init__(self, root=None, validate=True, absolute_paths=True,
                 validate=validate, absolute_paths=absolute_paths,
                 derivatives=None, sources=self, config=None,
                 regex_search=regex_search, reset_database=reset_database,
-                **indexer_kwargs)
+                indexer=indexer, **indexer_kwargs)
 
     @property
     def root(self):


Test patch:
Repository: bids-standard/pybids

Issue:
BIDSValidation error message doesn't provide path of erroneous `dataset_description.json` 
When creating a layout object or adding derivative directories, the [`BIDSDerivativesValidationError`](https://github.com/bids-standard/pybids/blob/f0d198da950371e64c6b2911627a549d78f62df3/bids/layout/validation.py#L130-L132) does not provide the file on which it errored out. This makes it hard for the user to debug the issue.


Discussion:

Resolving patch:
diff --git a/bids/layout/validation.py b/bids/layout/validation.py
--- a/bids/layout/validation.py
+++ b/bids/layout/validation.py
@@ -93,7 +93,8 @@ def validate_root(root, validate):
             if description is None:
                 raise BIDSValidationError(
                     "'dataset_description.json' is not a valid json file."
-                    " There is likely a typo in your 'dataset_description.json'."
+                    " There is likely a typo in your 'dataset_description.json' at "
+                    f"{target.resolve()}. "
                     "\nExample contents of 'dataset_description.json': \n%s" %
                     json.dumps(EXAMPLE_BIDS_DESCRIPTION)
                 ) from err
@@ -102,7 +103,8 @@ def validate_root(root, validate):
                 if k not in description:
                     raise BIDSValidationError(
                         "Mandatory %r field missing from "
-                        "'dataset_description.json'."
+                        "'dataset_description.json' at "
+                        f"{target.resolve()}. "
                         "\nExample: %s" % (k, MANDATORY_BIDS_FIELDS[k])
                     )
 
@@ -130,7 +132,8 @@ def validate_derivative_path(path, **kwargs):
         raise BIDSDerivativesValidationError(
             "Every valid BIDS-derivatives dataset must "
             "have a GeneratedBy.Name field set "
-            "inside 'dataset_description.json'. "
+            "inside 'dataset_description.json', "
+            f"here {dd.resolve()}. "
             f"\nExample: {MANDATORY_DERIVATIVES_FIELDS['GeneratedBy']}"
         )
     return pipeline_name


Test patch:
Repository: bids-standard/pybids

Issue:
UnboundLocalError when using BIDSLayout with specific indexer
Hello!

I encountered an UnboundLocalError when trying to create a BIDSLayout with a specific indexer. Here's the code snippet that produces the error:

```
import bids

filters = {'datatype': 'eeg', 'suffix': 'eeg'}
indexer = bids.BIDSLayoutIndexer(**filters)
layout = bids.BIDSLayout('../data/bids-examples/eeg_ds000117', indexer=indexer)
```

Here's the error message:

`UnboundLocalError: cannot access local variable 'ext_key' where it is not associated with a value`

The error occurs in the following lines of the PyBIDS package:

```
layout.py:177, in BIDSLayout.__init__
index.py:154, in BIDSLayoutIndexer.__call__
index.py:268, in BIDSLayoutIndexer._index_metadata
```

In the `_index_metadata` method, the variable `ext_key` is used in the `if filters.get(ext_key):` statement before it has been assigned a value. The error seems to occur because of this issue.

Environment:
- PyBIDS version: 0.16.4
- Python version: 3.12
- Operating System: macOS

I would appreciate any help or guidance on this issue. If you need any more information, please let me know. Thank you in advance for your time and assistance.


Discussion:

Resolving patch:
diff --git a/bids/layout/index.py b/bids/layout/index.py
--- a/bids/layout/index.py
+++ b/bids/layout/index.py
@@ -270,11 +270,11 @@ def _index_metadata(self):
             # ensure we are returning objects
             filters['return_type'] = 'object'
 
-            if filters.get(ext_key):
-                filters[ext_key] = listify(filters[ext_key])
+            if filters.get('extension'):
+                filters['extension'] = listify(filters['extension'])
                 # ensure json files are being indexed
-                if '.json' not in filters[ext_key]:
-                    filters[ext_key].append('.json')
+                if '.json' not in filters['extension']:
+                    filters['extension'].append('.json')
 
         # Process JSON files first if we're indexing metadata
         all_files = self._layout.get(absolute_paths=True, **filters)
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -139,7 +139,7 @@ def __init__(self, root=None, validate=True, absolute_paths=True,
         ]):
             try:
                 self.source_pipeline = validate_derivative_path(root)
-            except BIDSValidationError as err:
+            except (BIDSValidationError, FileNotFoundError) as err:
                 if validate:
                     raise err
                 self.source_pipeline = None


Test patch:
Repository: YunoHost/apps

Issue:
Simplify current version
As discuss at YunoHost Meeting 06/10/2022, remove the comment after the shipped version
Close #1522


Discussion:

Resolving patch:
diff --git a/tools/README-generator/make_readme.py b/tools/README-generator/make_readme.py
--- a/tools/README-generator/make_readme.py
+++ b/tools/README-generator/make_readme.py
@@ -32,7 +32,7 @@ def generate_READMEs(app_path: str):
     from_catalog = catalog.get(manifest['id'], {})
 
     antifeatures_list = yaml.load(open(Path(os.path.abspath(__file__)).parent.parent.parent / "antifeatures.yml"), Loader=yaml.SafeLoader)
-    antifeatures_list = {e['id']: e for e in antifeatures_list}
+    antifeatures_list = { e['id']: e for e in antifeatures_list }
 
     if not upstream and not (app_path / "doc" / "DISCLAIMER.md").exists():
         print(
@@ -69,18 +69,8 @@ def generate_READMEs(app_path: str):
         else:
             disclaimer = None
 
-        # Get the current branch using git inside the app path
-        default_branch = from_catalog.get('branch', 'master')
-        current_branch = os.popen(f"git -C {app_path} rev-parse --abbrev-ref HEAD").read().strip()
-
-        if default_branch != current_branch:
-            os.system(f"git -C {app_path} fetch origin {default_branch} 2>/dev/null")
-            default_branch_version = os.popen(f"git -C {app_path} show FETCH_HEAD:manifest.json | jq -r .version").read().strip()
-        else:
-            default_branch_version = None  # we don't care in that case
-
         # TODO: Add url to the documentation... and actually create that documentation :D
-        antifeatures = {a: antifeatures_list[a] for a in from_catalog.get('antifeatures', [])}
+        antifeatures = { a: antifeatures_list[a] for a in from_catalog.get('antifeatures', [])}
         for k, v in antifeatures.items():
             antifeatures[k]['title'] = value_for_lang(v['title'], lang_suffix)
             if manifest.get("antifeatures", {}).get(k, None):
@@ -96,9 +86,6 @@ def generate_READMEs(app_path: str):
             disclaimer=disclaimer,
             antifeatures=antifeatures,
             manifest=manifest,
-            current_branch=current_branch,
-            default_branch=default_branch,
-            default_branch_version=default_branch_version,
         )
         (app_path / f"README{lang_suffix}.md").write_text(out)
 


Test patch:
Repository: aws/sagemaker-python-sdk

Issue:
creating estimator with role name fails if role is default sagemaker service role
if you pass a role name (instead of full ARN) to estimator constructor, and you are using default role created by SageMaker, training job will fail with "unable to assume the role" ClientError:

`sagemaker-mxnet-py2-cpu-2018-01-14-15-35-12-859: Failed Reason: ClientError: SageMaker was unable to assume the role 'arn:aws:iam::<ACCOUNT>:role/AmazonSageMaker-ExecutionRole-20180114T073202'`

error message shows the sdk converted role name to ARN without the 'service-role' prefix. running the same job with corrected full ARN works:

`arn:aws:iam::<ACCOUNT>:role/service-role/AmazonSageMaker-ExecutionRole-20180114T075260`

i encountered this error starting a training job, but I assume it would impact endpoint creation in same way.





Discussion:
Thanks for bringing this issue to our attention. We're looking into it and will post correspondence here when we have an update. 

Thanks -- Owen

Resolving patch:
diff --git a/src/sagemaker/session.py b/src/sagemaker/session.py
--- a/src/sagemaker/session.py
+++ b/src/sagemaker/session.py
@@ -522,8 +522,8 @@ def endpoint_from_production_variants(self, name, production_variants, wait=True
     def expand_role(self, role):
         """Expand an IAM role name into an ARN.
 
-        If the role is already in the form of an ARN, then the role is simply returned. Otherwise, the role
-        is formatted as an ARN, using the current account as the IAM role's AWS account.
+        If the role is already in the form of an ARN, then the role is simply returned. Otherwise we retrieve the full
+        ARN and return it.
 
         Args:
             role (str): An AWS IAM role (either name or full ARN).
@@ -534,8 +534,7 @@ def expand_role(self, role):
         if '/' in role:
             return role
         else:
-            account = self.boto_session.client('sts').get_caller_identity()['Account']
-            return 'arn:aws:iam::{}:role/{}'.format(account, role)
+            return boto3.resource("iam").Role(role).arn
 
     def get_caller_identity_arn(self):
         """Returns the ARN user or role whose credentials are used to call the API.


Test patch:
Repository: sonic-net/sonic-utilities

Issue:
add config port Ethernet down/up 
add config cli to bring port up/down.


Discussion:

Resolving patch:
diff --git a/config/main.py b/config/main.py
--- a/config/main.py
+++ b/config/main.py
@@ -229,6 +229,39 @@ def neighbor(ipaddr_or_hostname, verbose):
 
     _bgp_session_startup(bgp_asn, ipaddress, verbose)
 
+#
+# 'interface' group
+#
+
+@cli.group()
+def interface():
+    """Interface-related configuration tasks"""
+    pass
+
+#
+# 'shutdown' subcommand
+#
+
+@interface.command()
+@click.argument('interface_name', metavar='<interface_name>', required=True)
+@click.option('-v', '--verbose', is_flag=True, help="Enable verbose output")
+def shutdown(interface_name, verbose):
+    """Shut down interface"""
+    command = "ip link set {} down".format(interface_name)
+    run_command(command, display_cmd=verbose)
+
+#
+# 'startup' subcommand
+#
+
+@interface.command()
+@click.argument('interface_name', metavar='<interface_name>', required=True)
+@click.option('-v', '--verbose', is_flag=True, help="Enable verbose output")
+def startup(interface_name, verbose):
+    """Start up interface"""
+    command = "ip link set {} up".format(interface_name)
+    run_command(command, display_cmd=verbose)
+
 
 if __name__ == '__main__':
     cli()


Test patch:
Repository: sonic-net/sonic-utilities

Issue:
[config] -h not supported for config
<!--
If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below. If you fail to provide this
information within 7 days, we cannot debug your issue and will close it. We
will, however, reopen it if you later provide the information.

For more information about reporting issues, see
https://github.com/Azure/SONiC/wiki#report-issues

---------------------------------------------------
GENERAL SUPPORT INFORMATION
---------------------------------------------------

The GitHub issue tracker is for bug reports and feature requests.
General support can be found at the following locations:

- SONiC Support Forums - https://groups.google.com/forum/#!forum/sonicproject

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

**Description**
-h does not print help information for config commands

<!--
Briefly describe the problem you are having in a few paragraphs.
-->

**Steps to reproduce the issue**
1. Enter 'sudo config -h'

**Describe the results you received**
CLI returns "Error: no such option: -h"

**Describe the results you expected**
CLI should print help information (e.g. same behavior as --help)

**Output of `show version`**
```
SONiC Software Version: SONiC.master.0-dirty-20190913.095551
Distribution: Debian 9.11
Kernel: 4.9.0-9-2-amd64
Build commit: d47ec526
Build date: Fri Sep 13 17:07:26 UTC 2019
Built by: daall@valhalla
```

<!--
Also attach debug file produced by `sudo generate_dump`
-->




Discussion:

Resolving patch:
diff --git a/config/main.py b/config/main.py
--- a/config/main.py
+++ b/config/main.py
@@ -18,6 +18,8 @@
 import aaa
 import mlnx
 
+CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help', '-?'])
+
 SONIC_CFGGEN_PATH = '/usr/local/bin/sonic-cfggen'
 SYSLOG_IDENTIFIER = "config"
 
@@ -386,8 +388,9 @@ def is_ipaddress(val):
         return False
     return True
 
+
 # This is our main entrypoint - the main 'config' command
-@click.group()
+@click.group(context_settings=CONTEXT_SETTINGS)
 def config():
     """SONiC command line - 'config' command"""
     if os.geteuid() != 0:


Test patch:
Repository: sonic-net/sonic-utilities

Issue:
Add filter for errors to "show interface counters"
**Description**
It would be helpful if one could run "`show interface counters errors`" to display a filtered version of show interface counters that includes only error counters (e.g. RX_ERR, RX_DRP, etc.).

**Describe the results you expected**
```
admin@str-s6000-on-5:~$ show interface counters errors
      IFACE    STATE    RX_ERR    RX_DRP    RX_OVR    TX_ERR    TX_DRP    TX_OVR
----------  ---------  --------  --------  -------- --------  --------  --------
  Ethernet0        U       200      1000         0       0         0         0
```



Discussion:

Resolving patch:
diff --git a/show/main.py b/show/main.py
--- a/show/main.py
+++ b/show/main.py
@@ -966,6 +966,28 @@ def counters(ctx, verbose, period, printall):
 
         run_command(cmd, display_cmd=verbose)
 
+# 'errors' subcommand ("show interfaces counters errors")
+@counters.command()
+@click.option('-p', '--period')
+@click.option('--verbose', is_flag=True, help="Enable verbose output")
+def errors(verbose, period):
+    """Show interface counters errors"""
+    cmd = "portstat -e"
+    if period is not None:
+        cmd += " -p {}".format(period)
+    run_command(cmd, display_cmd=verbose)
+
+# 'rates' subcommand ("show interfaces counters rates")
+@counters.command()
+@click.option('-p', '--period')
+@click.option('--verbose', is_flag=True, help="Enable verbose output")
+def rates(verbose, period):
+    """Show interface counters rates"""
+    cmd = "portstat -R"
+    if period is not None:
+        cmd += " -p {}".format(period)
+    run_command(cmd, display_cmd=verbose)
+
 # 'counters' subcommand ("show interfaces counters rif")
 @counters.command()
 @click.argument('interface', metavar='<interface_name>', required=False, type=str)


Test patch:
Repository: sonic-net/sonic-utilities

Issue:
 [PFCWD] The incorrect cli command doesn't return failed exit code 1
**Description**
The incorrect cli PFCWD command doesn't return failed exit code 1

**Steps to reproduce**

1) Run the cli command:
```
config pfcwd start --action drop ports all deteection-time 200 --restoration-time 200 ; echo AUTO_REMOTE_STATUS=$?
```

Observed behavior
```
root@r-qa-sw-eth-114:/home/admin# [sonic_dump_r-qa-sw-eth-2132_20210531_113146.tar.gz](https://github.com/Azure/sonic-buildimage/files/6569960/sonic_dump_r-qa-sw-eth-2132_20210531_113146.tar.gz)
Failed to run command, invalid options:
deteection-time



AUTO_REMOTE_STATUS=0
```

Expected behavior
```
root@r-qa-sw-eth-114:/home/admin# config pfcwd start --action drop ports all deteection-time 200 --restoration-time 200 ; echo AUTO_REMOTE_STATUS=$?
Failed to run command, invalid options:
deteection-time

AUTO_REMOTE_STATUS=1
```

**Output of show version**
```
SONiC Software Version: SONiC.202012.81-2fc748e_Internal
Distribution: Debian 10.9
Kernel: 4.19.0-12-2-amd64
Build commit: 2fc748ea
Build date: Thu May  6 20:42:29 UTC 2021
Built by: sw-r2d2-bot@r-build-sonic-ci02

Platform: x86_64-mlnx_msn3800-r0
HwSKU: ACS-MSN3800
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1925X00004
```
 [sonic_dump_r-qa-sw-eth-2132_20210531_113146.tar.gz](https://github.com/Azure/sonic-buildimage/files/6569960/sonic_dump_r-qa-sw-eth-2132_20210531_113146.tar.gz)


Discussion:
@lolyu please help to take a look. Thanks.
> @lolyu please help to take a look. Thanks.

Any update on that ? Thanks. 

Resolving patch:
diff --git a/pfcwd/main.py b/pfcwd/main.py
--- a/pfcwd/main.py
+++ b/pfcwd/main.py
@@ -242,7 +242,7 @@ def start(self, action, restoration_time, ports, detection_time):
             click.echo("Failed to run command, invalid options:")
             for opt in invalid_ports:
                 click.echo(opt)
-            exit()
+            exit(1)
         self.start_cmd(action, restoration_time, ports, detection_time)
 
 


Test patch:
diff --git a/tests/pfcwd_test.py b/tests/pfcwd_test.py
--- a/tests/pfcwd_test.py
+++ b/tests/pfcwd_test.py
@@ -255,7 +255,7 @@ def test_pfcwd_start_ports_invalid(self):
             obj=db
         )
         print(result.output)
-        assert result.exit_code == 0
+        assert result.exit_code == 1
         assert result.output == pfcwd_show_start_config_output_fail
 
     @classmethod
@@ -447,7 +447,7 @@ def test_pfcwd_start_ports_masic_invalid(self):
             obj=db
         )
         print(result.output)
-        assert result.exit_code == 0
+        assert result.exit_code == 1
         assert result.output == show_pfc_config_start_fail
 
         # get config after the command, config shouldn't change
@@ -500,4 +500,4 @@ def teardown_class(cls):
         import mock_tables.mock_single_asic
         importlib.reload(mock_tables.mock_single_asic)
         import pfcwd.main
-        importlib.reload(pfcwd.main)
\ No newline at end of file
+        importlib.reload(pfcwd.main)
Repository: aws/aws-sdk-pandas

Issue:
EMR Job with Docker image always use the deploy mode as cluster
### Describe the bug

Calling the submit_emr_job function
```python
wr.emr.submit_spark_step(
    cluster_id,
    s3_path,
    deploy_mode="client",
    docker_image=image_path
)
```

Always yield deploy_mode as cluster despite explicitly mentioning otherwise

### How to Reproduce

```python
wr.emr.submit_spark_step(
    cluster_id,
    s3_path,
    deploy_mode="client",
    docker_image=image_path
)
```


### Expected behavior

Deploy mode should be client

### Your project

_No response_

### Screenshots

_No response_

### OS

Mac

### Python version

3.9.9

### AWS DataWrangler version

2.15

### Additional context

_No response_


Discussion:

Resolving patch:
diff --git a/awswrangler/emr.py b/awswrangler/emr.py
--- a/awswrangler/emr.py
+++ b/awswrangler/emr.py
@@ -1022,7 +1022,7 @@ def build_spark_step(
     else:
         config: str = "hdfs:///user/hadoop/config.json"
         cmd = (
-            f"spark-submit --deploy-mode cluster "
+            f"spark-submit --deploy-mode {deploy_mode} "
             f"--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker "
             f"--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE={docker_image} "
             f"--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG={config} "


Test patch: