internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: zestedesavoir/zds-site

Issue:
La connexion avec Google ne fonctionne pas sur le serveur de bêta
Problème rapporté par Sentry.

La connexion avec Google ne fonctionne pas sur le serveur de bêta, on a une erreur 500 à cause d'une erreur dans le paquet social-auth-core. Même en testant avec la dernière version de ce paquet le problème n'est pas corrigé.

J'ai ouvert un ticket chez social-auth-core : https://github.com/python-social-auth/social-core/issues/847



Discussion:

Resolving patch:
diff --git a/zds/settings/abstract_base/requirements.py b/zds/settings/abstract_base/requirements.py
--- a/zds/settings/abstract_base/requirements.py
+++ b/zds/settings/abstract_base/requirements.py
@@ -15,17 +15,17 @@
 SOCIAL_AUTH_FACEBOOK_SCOPE = ["email"]
 
 SOCIAL_AUTH_PIPELINE = (
-    "social.pipeline.social_auth.social_details",
-    "social.pipeline.social_auth.social_uid",
-    "social.pipeline.social_auth.auth_allowed",
-    "social.pipeline.social_auth.social_user",
-    "social.pipeline.user.get_username",
-    "social.pipeline.social_auth.associate_by_email",
-    "social.pipeline.user.create_user",
+    "social_core.pipeline.social_auth.social_details",
+    "social_core.pipeline.social_auth.social_uid",
+    "social_core.pipeline.social_auth.auth_allowed",
+    "social_core.pipeline.social_auth.social_user",
+    "social_core.pipeline.user.get_username",
+    "social_core.pipeline.social_auth.associate_by_email",
+    "social_core.pipeline.user.create_user",
     "zds.member.models.save_profile",
-    "social.pipeline.social_auth.associate_user",
-    "social.pipeline.social_auth.load_extra_data",
-    "social.pipeline.user.user_details",
+    "social_core.pipeline.social_auth.associate_user",
+    "social_core.pipeline.social_auth.load_extra_data",
+    "social_core.pipeline.user.user_details",
 )
 
 # Before adding new providers such as Facebook and Google,
diff --git a/zds/settings/prod.py b/zds/settings/prod.py
--- a/zds/settings/prod.py
+++ b/zds/settings/prod.py
@@ -122,23 +122,6 @@ def _get_version():
 }
 
 
-# python-social-auth
-# http://psa.matiasaguirre.net/docs/configuration/django.html
-SOCIAL_AUTH_PIPELINE = (
-    "social.pipeline.social_auth.social_details",
-    "social.pipeline.social_auth.social_uid",
-    "social.pipeline.social_auth.auth_allowed",
-    "social.pipeline.social_auth.social_user",
-    "social.pipeline.user.get_username",
-    "social.pipeline.social_auth.associate_by_email",
-    "social.pipeline.user.create_user",
-    "zds.member.models.save_profile",
-    "social.pipeline.social_auth.associate_user",
-    "social.pipeline.social_auth.load_extra_data",
-    "social.pipeline.user.user_details",
-)
-
-
 ###############################################################################
 # ZESTE DE SAVOIR SETTINGS
 


Test patch:
Repository: spotify/luigi

Issue:
LuigiConfigParser::add_config_path() raises if instance() hasn't been accessed
To add a path to the list of config paths, one currently has to do:

``` python
LuigiConfigParser.instance()  # remove this and get an exception
LuigiConfigParser.add_config_path(my_path)
```

because `add_config_path` tries to reload `cls._instance` which is initialized with `None`. Wouldn't it be cleaner to do a check there and only reload a non-null instance?



Discussion:
Sound good, feel free to submit a PR!


Resolving patch:
diff --git a/luigi/configuration.py b/luigi/configuration.py
--- a/luigi/configuration.py
+++ b/luigi/configuration.py
@@ -14,7 +14,7 @@ class LuigiConfigParser(ConfigParser):
     @classmethod
     def add_config_path(cls, path):
         cls._config_paths.append(path)
-        cls._instance.reload()
+        cls.reload()
 
     @classmethod
     def instance(cls, *args, **kwargs):
@@ -26,8 +26,9 @@ def instance(cls, *args, **kwargs):
 
         return cls._instance
 
-    def reload(self):
-        return self._instance.read(self._config_paths)
+    @classmethod
+    def reload(cls):
+        return cls.instance().read(cls._config_paths)
 
     def _get_with_default(self, method, section, option, default, expected_type=None):
         """ Gets the value of the section/option using method. Returns default if value


Test patch:
Repository: spotify/luigi

Issue:
hardcoded unix tyle tmp directory cause failure on Windows
The culprit line is: https://github.com/spotify/luigi/blob/master/luigi/interface.py#L92

```
s_global=True, default='/var/tmp/luigi',
```

needs to be something like:

```
is_global=True, default=tempfile.gettempdir()+os.sep+'luigi',
```

after importing `tempfile` of course.

workaround - manually create directory `\var\tmp\luigi`



Discussion:
I think you can do `default=os.path.join(tempfile.gettempdir(), 'luigi')`

Feel free to put together a PR and submit


Resolving patch:
diff --git a/luigi/interface.py b/luigi/interface.py
--- a/luigi/interface.py
+++ b/luigi/interface.py
@@ -27,6 +27,7 @@
 import argparse
 import sys
 import os
+import tempfile
 from task import Register
 
 
@@ -79,7 +80,7 @@ class EnvironmentParamsContainer(task.Task):
         is_global=True, default=False,
         description='Ignore if similar process is already running')
     lock_pid_dir = parameter.Parameter(
-        is_global=True, default='/var/tmp/luigi',
+        is_global=True, default=os.path.join(tempfile.gettempdir(), 'luigi'),
         description='Directory to store the pid file')
     workers = parameter.IntParameter(
         is_global=True, default=1,


Test patch:
Repository: spotify/luigi

Issue:
bug on luigi.contrib.ftp.AtomicFtpfile
on luigi.contrib.ftp.AtomicFtpfile, self.path is accessed before being set.

```
luigi/contrib/ftp.py Line 170 Character 26
167:    """
168:    
169:    def __init__(self, fs, path):
170:    self.__tmp_path = self.path + '-luigi-tmp-%09d' % random.randrange(0, 1e10)
    Access to member 'path' before its definition line 172
171:    self._fs = fs
172:    self.path = path
173:    super(AtomicFtpfile, self).__init__(self.__tmp_path, 'w')
```

captured by [landscape.io](https://landscape.io/github/steenzout/luigi/4/messages/error)



Discussion:

Resolving patch:
diff --git a/luigi/contrib/ftp.py b/luigi/contrib/ftp.py
--- a/luigi/contrib/ftp.py
+++ b/luigi/contrib/ftp.py
@@ -167,7 +167,14 @@ class AtomicFtpfile(file):
     """
 
     def __init__(self, fs, path):
-        self.__tmp_path = self.path + '-luigi-tmp-%09d' % random.randrange(0, 1e10)
+        """
+        Initializes an AtomicFtpfile instance.
+
+        :param fs:
+        :param path:
+        :type path: str
+        """
+        self.__tmp_path = '%s-luigi-tmp-%09d' % (path, random.randrange(0, 1e10))
         self._fs = fs
         self.path = path
         super(AtomicFtpfile, self).__init__(self.__tmp_path, 'w')


Test patch:
Repository: spotify/luigi

Issue:
hadoop --config cannot be set in client.cfg
We use a config option for hadoop command, so I put the following line to client.cfg:
command: /usr/bin/hadoop --config /etc/rutarget
This results in a "no such file" error raised by Popen.

Here is a simple [fix](https://github.com/kalaidin/luigi/commit/f9db4c189e5171812f959a19d516ca51850bc5fa) I can provide, not sure if this is a right way to go.



Discussion:

Resolving patch:
diff --git a/luigi/hdfs.py b/luigi/hdfs.py
--- a/luigi/hdfs.py
+++ b/luigi/hdfs.py
@@ -52,7 +52,7 @@ def call_check(command):
 
 
 def load_hadoop_cmd():
-    return [luigi.configuration.get_config().get('hadoop', 'command', 'hadoop')]
+    return luigi.configuration.get_config().get('hadoop', 'command', 'hadoop').split()
 
 
 def tmppath(path=None, include_unix_username=True):
diff --git a/luigi/tools/range.py b/luigi/tools/range.py
--- a/luigi/tools/range.py
+++ b/luigi/tools/range.py
@@ -81,7 +81,7 @@ class RangeBase(luigi.WrapperTask):
     # define here generically without dark magic. Refer to the overrides.
     start = luigi.Parameter()
     stop = luigi.Parameter()
-    reverse = luigi.BooleanParameter(
+    reverse = luigi.BoolParameter(
         default=False,
         description="specifies the preferred order for catching up. False - work from the oldest missing outputs onward; True - from the newest backward")
     task_limit = luigi.IntParameter(


Test patch:
Repository: spotify/luigi

Issue:
Latest Luigi version breaks my hadoop jobs
I used to be able to initialize a class like

```
class MyStreamingJob(luigi.hadoop.JobTask):
    param = luigi.Parameter()
```

with code like

```
MyStreamingJob('param_value')
```

but now this results in an error

```
luigi.parameter.MissingParameterException: MyStreamingJob[args=('param_value',), kwargs={}]: requires the 'param' parameter to be set
```

I believe it's now interpreting the first argument as pool, where it never did that before.



Discussion:
Thanks – this seems to be the case. I don't know why our existing tests aren't catching it. Let me try to add a test + fix


Resolving patch:
diff --git a/luigi/task.py b/luigi/task.py
--- a/luigi/task.py
+++ b/luigi/task.py
@@ -326,7 +326,7 @@ def get_param_values(cls, params, args, kwargs):
         exc_desc = '%s[args=%s, kwargs=%s]' % (task_name, args, kwargs)
 
         # Fill in the positional arguments
-        positional_params = [(n, p) for n, p in params]
+        positional_params = [(n, p) for n, p in params if p.significant]
         for i, arg in enumerate(args):
             if i >= len(positional_params):
                 raise parameter.UnknownParameterException('%s: takes at most %d parameters (%d given)' % (exc_desc, len(positional_params), len(args)))


Test patch:
diff --git a/test/hadoop_test.py b/test/hadoop_test.py
--- a/test/hadoop_test.py
+++ b/test/hadoop_test.py
@@ -155,6 +155,10 @@ def output(self):
         return self.get_output('failing')
 
 
+class MyStreamingJob(luigi.hadoop.JobTask):
+    param = luigi.Parameter()
+
+
 def read_wordcount_output(p):
     count = {}
     for line in p.open('r'):
@@ -228,6 +232,10 @@ def test_unicode_job(self):
     def test_failing_job(self):
         CommonTests.test_failing_job(self)
 
+    def test_instantiate_job(self):
+        # See https://github.com/spotify/luigi/issues/738
+        MyStreamingJob('param_value')
+
     def setUp(self):
         MockFile.fs.clear()
 
diff --git a/test/parameter_test.py b/test/parameter_test.py
--- a/test/parameter_test.py
+++ b/test/parameter_test.py
@@ -273,11 +273,16 @@ def test_global_param_shared(self):
 
     def test_insignificant_parameter(self):
         class InsignificantParameterTask(luigi.Task):
-            foo = luigi.Parameter(significant=False)
+            foo = luigi.Parameter(significant=False, default='foo_default')
             bar = luigi.Parameter()
 
-        t = InsignificantParameterTask(foo='x', bar='y')
-        self.assertEqual(t.task_id, 'InsignificantParameterTask(bar=y)')
+        t1 = InsignificantParameterTask(foo='x', bar='y')
+        self.assertEqual(t1.task_id, 'InsignificantParameterTask(bar=y)')
+
+        t2 = InsignificantParameterTask('z')
+        self.assertEqual(t2.foo, 'foo_default')
+        self.assertEqual(t2.bar, 'z')
+        self.assertEqual(t2.task_id, 'InsignificantParameterTask(bar=z)')
 
 
 class TestNewStyleGlobalParameters(unittest.TestCase):
Repository: spotify/luigi

Issue:
`S3FlagTarget` checks for slash terminated path with `is` instead of `==`
See https://github.com/spotify/luigi/blob/master/luigi/s3.py#L510

This screws up situations where the provided data may be a unicode string.  Is this deliberate?



Discussion:

Resolving patch:
diff --git a/luigi/s3.py b/luigi/s3.py
--- a/luigi/s3.py
+++ b/luigi/s3.py
@@ -507,7 +507,7 @@ def __init__(self, path, format=None, client=None, flag='_SUCCESS'):
         :param flag:
         :type flag: str
         """
-        if path[-1] is not "/":
+        if path[-1] != "/":
             raise ValueError("S3FlagTarget requires the path to be to a "
                              "directory.  It must end with a slash ( / ).")
         super(S3Target, self).__init__(path)


Test patch:
Repository: spotify/luigi

Issue:
specifying ssh port for RemoteTarget
It would be nice to be able to specify a port number when establishing a RemoteTarget.



Discussion:
Want to create a PR for it? :)


Resolving patch:
diff --git a/luigi/contrib/ssh.py b/luigi/contrib/ssh.py
--- a/luigi/contrib/ssh.py
+++ b/luigi/contrib/ssh.py
@@ -48,15 +48,16 @@
 
 class RemoteContext(object):
 
-    def __init__(self, host, username=None, key_file=None, connect_timeout=None):
+    def __init__(self, host, username=None, key_file=None, connect_timeout=None, port=None):
         self.host = host
         self.username = username
         self.key_file = key_file
         self.connect_timeout = connect_timeout
+        self.port = port
 
     def __repr__(self):
-        return '%s(%r, %r, %r, %r)' % (
-            type(self).__name__, self.host, self.username, self.key_file, self.connect_timeout)
+        return '%s(%r, %r, %r, %r, %r)' % (
+            type(self).__name__, self.host, self.username, self.key_file, self.connect_timeout, self.port)
 
     def __eq__(self, other):
         return repr(self) == repr(other)
@@ -75,6 +76,8 @@ def _prepare_cmd(self, cmd):
                           "-S", "none",  # disable ControlMaster since it causes all sorts of weird behaviour with subprocesses...
                           "-o", "BatchMode=yes",  # no password prompts etc
                           ]
+        if self.port:
+            connection_cmd.extend(["-p", self.port])
 
         if self.connect_timeout is not None:
             connection_cmd += ['-o', 'ConnectTimeout=%d' % self.connect_timeout]
@@ -127,8 +130,8 @@ def tunnel(self, local_port, remote_port=None, remote_host="localhost"):
 
 class RemoteFileSystem(luigi.target.FileSystem):
 
-    def __init__(self, host, username=None, key_file=None):
-        self.remote_context = RemoteContext(host, username, key_file)
+    def __init__(self, host, username=None, key_file=None, port=None):
+        self.remote_context = RemoteContext(host, username, key_file, port)
 
     def exists(self, path):
         """
@@ -158,6 +161,8 @@ def _scp(self, src, dest):
         cmd = ["scp", "-q", "-B", "-C", "-o", "ControlMaster=no"]
         if self.remote_context.key_file:
             cmd.extend(["-i", self.remote_context.key_file])
+        if self.remote_context.port:
+            cmd.extend(["-P", self.remote_context.port])
         cmd.extend([src, dest])
         p = subprocess.Popen(cmd)
         output, _ = p.communicate()
@@ -228,12 +233,12 @@ class RemoteTarget(luigi.target.FileSystemTarget):
     The target is implemented using ssh commands streaming data over the network.
     """
 
-    def __init__(self, path, host, format=None, username=None, key_file=None):
+    def __init__(self, path, host, format=None, username=None, key_file=None, port=None):
         super(RemoteTarget, self).__init__(path)
         if format is None:
             format = luigi.format.get_default_format()
         self.format = format
-        self._fs = RemoteFileSystem(host, username, key_file)
+        self._fs = RemoteFileSystem(host, username, key_file, port)
 
     @property
     def fs(self):


Test patch:
Repository: spotify/luigi

Issue:
RemoteFileSystem uses wrong path separator
The RemoteFileSystem uses the client path  separators instead of the remote system path separators.

That means that when trying to write to a Linux RemoteTarget from a Windows Client, the path will be broken (backslashes instead of slashes as path separators, leading to wrong path names).



Discussion:
To clarify, the problem is in RemoteFileSystem.put

A simple fix would be to make the path separator configurable.


Resolving patch:
diff --git a/luigi/contrib/ssh.py b/luigi/contrib/ssh.py
--- a/luigi/contrib/ssh.py
+++ b/luigi/contrib/ssh.py
@@ -40,6 +40,7 @@
 import os
 import random
 import subprocess
+import posixpath
 
 import luigi
 import luigi.format
@@ -173,7 +174,7 @@ def _scp(self, src, dest):
 
     def put(self, local_path, path):
         # create parent folder if not exists
-        normpath = os.path.normpath(path)
+        normpath = posixpath.normpath(path)
         folder = os.path.dirname(normpath)
         if folder and not self.exists(folder):
             self.remote_context.check_output(['mkdir', '-p', folder])


Test patch:
Repository: spotify/luigi

Issue:
hive table_exists should be case insensitive?
Any thoughts on this one?

In https://github.com/spotify/luigi/blob/master/luigi/contrib/hive.py#L141
(possibly here too, but we're not on CDH: https://github.com/spotify/luigi/blob/master/luigi/contrib/hive.py#L192)

Sometimes we have tables that are defined as capitalized, rather than lower case underscored names.  These are easier to read in code if left capitalized, though hive is case insensitive, and will return them as lower case.

E.g. when checking for an existing table `table = 'FooBar'`, stdout will return with `foobar` and the test will fail

This wasn't an issue in older versions, which just checked for string "does not exist" or "Table not found" in stdout.

Would be easy to fix using `return stdout and table.lower() in stdout` or `return stdout and table.lower() in stdout.lower()`

Let me know your thoughts on this.  I can supply a pull request if necessary.

Thanks,
Lin.



Discussion:
Please do give a pull request, with an accompanying test if possible :)


Resolving patch:
diff --git a/luigi/contrib/hive.py b/luigi/contrib/hive.py
--- a/luigi/contrib/hive.py
+++ b/luigi/contrib/hive.py
@@ -138,7 +138,7 @@ def table_exists(self, table, database='default', partition=None):
         if partition is None:
             stdout = run_hive_cmd('use {0}; show tables like "{1}";'.format(database, table))
 
-            return stdout and table in stdout
+            return stdout and table.lower() in stdout
         else:
             stdout = run_hive_cmd("""use %s; show partitions %s partition
                                 (%s)""" % (database, table, self.partition_spec(partition)))


Test patch:
diff --git a/test/contrib/hive_test.py b/test/contrib/hive_test.py
--- a/test/contrib/hive_test.py
+++ b/test/contrib/hive_test.py
@@ -106,6 +106,10 @@ def test_table_exists(self, run_command):
         returned = self.client.table_exists("mytable")
         self.assertTrue(returned)
 
+        # Issue #896 test case insensitivity
+        returned = self.client.table_exists("MyTable")
+        self.assertTrue(returned)
+
         run_command.return_value = "day=2013-06-28/hour=3\n" \
                                    "day=2013-06-28/hour=4\n" \
                                    "day=2013-07-07/hour=2\n"
@@ -166,6 +170,10 @@ def test_apacheclient_table_exists(self, run_command):
         returned = self.apacheclient.table_exists("mytable")
         self.assertTrue(returned)
 
+        # Issue #896 test case insensitivity
+        returned = self.apacheclient.table_exists("MyTable")
+        self.assertTrue(returned)
+
         run_command.return_value = "day=2013-06-28/hour=3\n" \
                                    "day=2013-06-28/hour=4\n" \
                                    "day=2013-07-07/hour=2\n"
Repository: spotify/luigi

Issue:
Failed to import 'db_task_history'
I tried enabling task history but it fails with:

```
Traceback (most recent call last):
  File "/home/danderson/bin/luigid", line 9, in <module>
load_entry_point('luigi==1.1.3', 'console_scripts', 'luigid')()
  File "/home/danderson/lib/python3.4/site-packages/luigi-1.1.3-py3.4.egg/luigi/cmdline.py", line 36, in luigid
logdir=opts.logdir)
  File "/home/danderson/lib/python3.4/site-packages/luigi-1.1.3-py3.4.egg/luigi/process.py", line 117, in daemonize
cmd(api_port=api_port, address=address)
  File "/home/danderson/lib/python3.4/site-packages/luigi-1.1.3-py3.4.egg/luigi/server.py", line 254, in run
scheduler = CentralPlannerScheduler()
  File "/home/danderson/lib/python3.4/site-packages/luigi-1.1.3-py3.4.egg/luigi/scheduler.py", line 452, in __init__
import db_task_history  # Needs sqlalchemy, thus imported here
ImportError: No module named 'db_task_history'
```

I checked and db_task_history.py is where you would expect it to be



Discussion:

Resolving patch:
diff --git a/luigi/scheduler.py b/luigi/scheduler.py
--- a/luigi/scheduler.py
+++ b/luigi/scheduler.py
@@ -471,7 +471,7 @@ def __init__(self, config=None, resources=None, task_history_impl=None, **kwargs
         if task_history_impl:
             self._task_history = task_history_impl
         elif self._config.record_task_history:
-            import db_task_history  # Needs sqlalchemy, thus imported here
+            from luigi import db_task_history  # Needs sqlalchemy, thus imported here
             self._task_history = db_task_history.DbTaskHistory()
         else:
             self._task_history = history.NopHistory()


Test patch:
Repository: spotify/luigi

Issue:
gcs._DeleteOnCloseFile().close() cannot be called more than once
The issue is that [`close()`](https://github.com/spotify/luigi/blob/master/luigi/contrib/gcs.py#L371-L373) does not to check to see if the file exists before trying to delete, so an exception is raised if the file doesn't exist.



Discussion:

Resolving patch:
diff --git a/luigi/contrib/gcs.py b/luigi/contrib/gcs.py
--- a/luigi/contrib/gcs.py
+++ b/luigi/contrib/gcs.py
@@ -370,7 +370,12 @@ def download(self, path, chunksize=None):
 class _DeleteOnCloseFile(io.FileIO):
     def close(self):
         super(_DeleteOnCloseFile, self).close()
-        os.remove(self.name)
+        try:
+            os.remove(self.name)
+        except OSError:
+            # Catch a potential threading race condition and also allow this
+            # method to be called multiple times.
+            pass
 
     def readable(self):
         return True


Test patch:
diff --git a/test/contrib/gcs_test.py b/test/contrib/gcs_test.py
--- a/test/contrib/gcs_test.py
+++ b/test/contrib/gcs_test.py
@@ -159,3 +159,19 @@ class GCSTargetTest(_GCSBaseTestCase, FileSystemTargetTestMixin):
 
     def create_target(self, format=None):
         return gcs.GCSTarget(bucket_url(self.id()), format=format, client=self.client)
+
+    def test_close_twice(self):
+        # Ensure gcs._DeleteOnCloseFile().close() can be called multiple times
+        tgt = self.create_target()
+
+        with tgt.open('w') as dst:
+            dst.write('data')
+        assert dst.closed
+        dst.close()
+        assert dst.closed
+
+        with tgt.open() as src:
+            assert src.read().strip() == 'data'
+        assert src.closed
+        src.close()
+        assert src.closed
Repository: spotify/luigi

Issue:
Add method to S3Target to get back path
With an `S3Target(path)`, we can load data into S3 at the specified path. It would be useful if we could also get back the path at which the target is pointing to so it can be used in subsequent tasks.

Correct me if I'm wrong, but I couldn't find any way to do this. I can work on a PR if this sounds reasonable.



Discussion:
Yes we should just set `self.path = path` in the constructor of `S3Target` :)


Resolving patch:
diff --git a/luigi/s3.py b/luigi/s3.py
--- a/luigi/s3.py
+++ b/luigi/s3.py
@@ -501,6 +501,7 @@ def __init__(self, path, format=None, client=None):
         if format is None:
             format = get_default_format()
 
+        self.path = path
         self.format = format
         self.fs = client or S3Client()
 


Test patch:
diff --git a/test/s3_test.py b/test/s3_test.py
--- a/test/s3_test.py
+++ b/test/s3_test.py
@@ -100,6 +100,11 @@ def test_read_iterator_long(self):
         self.assertEqual("line two" + os.linesep, lines[1])
         self.assertEqual("line three", lines[2])
 
+    def test_get_path(self):
+        t = self.create_target()
+        path = t.path
+        self.assertEqual('s3://mybucket/test_file', path)
+
 
 class TestS3Client(unittest.TestCase):