internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: spotify/luigi

Issue:
Scheduler only hosts on unix socket when run in the background
Support for hosting the central scheduler on a unix socket was added, which is nice, but the scheduler ignores the `--unix-socket` argument from the command line when `--background` is not also supplied.  

This will work properly, and the scheduler will listen on the provided unix socket:

```
luigid --unix-socket /path/to/socket --background
```

With this command, the scheduler will still listen on the default port (8082):

```
luigid --unix-socket /path/to/socket
```

Fixing this would be a simple matter of passing the `unix_socket` argument onto the call to `server.run` in the case where the server is not daemonized, but was there a reason this functionality was left out in the first place?  If so, it probably ought to be in the documentation; as is, reading it gives me the impression that I should be able to tell the scheduler to listen on a unix socket regardless of whether it's running in the background.



Discussion:
@graingert maybe knows?

I've only ever run this in the background so never noticed

Oh ok. @jasonnance, would you like to send a PR with a fix? :)


Resolving patch:
diff --git a/luigi/cmdline.py b/luigi/cmdline.py
--- a/luigi/cmdline.py
+++ b/luigi/cmdline.py
@@ -41,4 +41,4 @@ def luigid(argv=sys.argv[1:]):
                                 filename=os.path.join(opts.logdir, "luigi-server.log"))
         else:
             logging.basicConfig(level=logging.INFO, format=luigi.process.get_log_format())
-        luigi.server.run(api_port=opts.port, address=opts.address)
+        luigi.server.run(api_port=opts.port, address=opts.address, unix_socket=opts.unix_socket)


Test patch:
Repository: spotify/luigi

Issue:
Python 3.5 support
Luigi may already work with Python 3.5, but since the README doesn't mention it I thought I'd ask.

Does Luigi support Python 3.5?



Discussion:
Not sure – I haven't tried :)

If Travis supports Python 3.5, we can enable it and try!

Looks like [Travis does support Python 3.5](https://docs.travis-ci.com/user/languages/python/#Choosing-Python-versions-to-test-against). Nice... they even have a nightly build of Python!

``` yaml
language: python
python:
  - "2.6"
  - "2.7"
  - "3.2"
  - "3.3"
  - "3.4"
  - "3.5"
  - "3.5-dev" # 3.5 development branch
  - "nightly" # currently points to 3.6-dev
```

do you want to send a PR? we can see what happens :)


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -84,6 +84,7 @@ def get_static_files(path):
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
+        'Programming Language :: Python :: 3.5',
         'Topic :: System :: Monitoring',
     ],
 )


Test patch:
diff --git a/test/worker_parallel_scheduling_test.py b/test/worker_parallel_scheduling_test.py
--- a/test/worker_parallel_scheduling_test.py
+++ b/test/worker_parallel_scheduling_test.py
@@ -149,7 +149,7 @@ def test_raise_unpicklable_exception_in_complete(self, send):
             UnpicklableExceptionTask().complete()
         except Exception as e:
             ex = e
-        self.assertRaises(pickle.PicklingError, pickle.dumps, ex)
+        self.assertRaises(Exception, pickle.dumps, ex)
 
         # verify this can run async
         self.w.add(UnpicklableExceptionTask(), multiprocess=True)
Repository: spotify/luigi

Issue:
send_email doesn't trim recipient's email address
The send_email function simply split email address list by comma, and this leads to email address like " john.doe@example.com" (a space before the address).

The problem line is [here](https://github.com/spotify/luigi/blob/master/luigi/notifications.py#L260)

Suggest fix is:

``` python
for r in recipients:
    recipients_tmp.extend(r.split(',').strip())
```



Discussion:
Can you send a pull request? Thanks

Sure

great :)

@erikbern Ready for review.


Resolving patch:
diff --git a/luigi/notifications.py b/luigi/notifications.py
--- a/luigi/notifications.py
+++ b/luigi/notifications.py
@@ -257,7 +257,7 @@ def send_email(subject, message, sender, recipients, image_png=None):
     # separated in luigi.cfg
     recipients_tmp = []
     for r in recipients:
-        recipients_tmp.extend(r.split(','))
+        recipients_tmp.extend([a.strip() for a in r.split(',') if a.strip()])
 
     # Replace original recipients with the clean list
     recipients = recipients_tmp


Test patch:
Repository: spotify/luigi

Issue:
Terminal Width affects Lock identification under Centos 6.5
The luigi.lock.getpcmd function will return a shorter command line if the terminal is smaller in width. This can result in locks being misidentified as identical and can be a significant problem if your binary/scripts are located in deep paths.

Presumably there is some option that can be passed to ps to resolve this.

edit: It looks like if the ps command is changed to `ps x -wwo pid,args` that should take care of it.



Discussion:

Resolving patch:
diff --git a/luigi/lock.py b/luigi/lock.py
--- a/luigi/lock.py
+++ b/luigi/lock.py
@@ -42,7 +42,7 @@ def getpcmd(pid):
                 _, val = lines
                 return val
     else:
-        cmd = 'ps -xo pid,args'
+        cmd = 'ps x -wwo pid,args'
         with os.popen(cmd, 'r') as p:
             # Skip the column titles
             p.readline()


Test patch:
Repository: spotify/luigi

Issue:
Superfluos project_id parameter in BigQuery job definition
The parameter `project_id` of most tasks in `luigi.contrib.bigquery` is superfluous as per https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs. Indeed, if set to arbitrary values jobs still run successfully.

It doesn't really *hurt* but can be confusing at first sight as it suggest some additional dependency.



Discussion:

Resolving patch:
diff --git a/luigi/contrib/bigquery.py b/luigi/contrib/bigquery.py
--- a/luigi/contrib/bigquery.py
+++ b/luigi/contrib/bigquery.py
@@ -348,7 +348,6 @@ def copy(self,
         """
 
         job = {
-            "projectId": dest_table.project_id,
             "configuration": {
                 "copy": {
                     "sourceTable": {
@@ -513,7 +512,6 @@ def run(self):
         assert all(x.startswith('gs://') for x in source_uris)
 
         job = {
-            'projectId': output.table.project_id,
             'configuration': {
                 'load': {
                     'destinationTable': {
@@ -599,7 +597,6 @@ def run(self):
         logger.info('Query SQL: %s', query)
 
         job = {
-            'projectId': output.table.project_id,
             'configuration': {
                 'query': {
                     'query': query,


Test patch:
Repository: spotify/luigi

Issue:
Allow schema to be specified for target table in sqla.CopyToTable
This could be supported by passing overriding the `schema` attribute of the task, and passing this to the `sqlalchemy.MetaData` constructor in `sqla.CopyToTable.create_table`.

A consideration: should we also allow a schema to be specified for the marker table? My current use case requires them to be different.

If this sounds OK I'll write it up later and create a PR for review.


Discussion:
@sd2k Any update here?
@dlstadther I've been using a subclass of `CopyToTable` which overrides `create_table`. I'll send a PR.

Resolving patch:
diff --git a/luigi/contrib/sqla.py b/luigi/contrib/sqla.py
--- a/luigi/contrib/sqla.py
+++ b/luigi/contrib/sqla.py
@@ -273,6 +273,8 @@ class CopyToTable(luigi.Task):
     Usage:
 
     * subclass and override the required `connection_string`, `table` and `columns` attributes.
+    * optionally override the `schema` attribute to use a different schema for
+      the target table.
     """
     _logger = logging.getLogger('luigi-interface')
 
@@ -300,6 +302,11 @@ def table(self):
     # completely ignore the columns. Instead set the reflect value to True below
     columns = []
 
+    # Specify the database schema of the target table, if supported by the
+    # RDBMS. Note that this doesn't change the schema of the marker table.
+    # The schema MUST already exist in the database, or this will task fail.
+    schema = ''
+
     # options
     column_separator = "\t"  # how columns are separated in the file copied into postgres
     chunk_size = 5000   # default chunk size for insert
@@ -328,15 +335,21 @@ def construct_sqla_columns(columns):
         else:
             # if columns is specified as (name, type) tuples
             with engine.begin() as con:
-                metadata = sqlalchemy.MetaData()
+
+                if self.schema:
+                    metadata = sqlalchemy.MetaData(schema=self.schema)
+                else:
+                    metadata = sqlalchemy.MetaData()
+
                 try:
-                    if not con.dialect.has_table(con, self.table):
+                    if not con.dialect.has_table(con, self.table, self.schema or None):
                         sqla_columns = construct_sqla_columns(self.columns)
                         self.table_bound = sqlalchemy.Table(self.table, metadata, *sqla_columns)
                         metadata.create_all(engine)
                     else:
-                        metadata.reflect(only=[self.table], bind=engine)
-                        self.table_bound = metadata.tables[self.table]
+                        full_table = '.'.join([self.schema, self.table]) if self.schema else self.table
+                        metadata.reflect(only=[full_table], bind=engine)
+                        self.table_bound = metadata.tables[full_table]
                 except Exception as e:
                     self._logger.exception(self.table + str(e))
 


Test patch:
Repository: spotify/luigi

Issue:
Getting rid of repetitive "Starting pruning of task graph" / "Done pruning task graph" log entries
Is there a way to get rid of the repetitive `Starting pruning of task graph` / `Done pruning task graph` log entries? Or is there a reason they show up, about 1 per second (on 16core / 16 workers set up)?


Discussion:

Resolving patch:
diff --git a/luigi/scheduler.py b/luigi/scheduler.py
--- a/luigi/scheduler.py
+++ b/luigi/scheduler.py
@@ -691,11 +691,11 @@ def dump(self):
 
     @rpc_method()
     def prune(self):
-        logger.info("Starting pruning of task graph")
+        logger.debug("Starting pruning of task graph")
         self._prune_workers()
         self._prune_tasks()
         self._prune_emails()
-        logger.info("Done pruning task graph")
+        logger.debug("Done pruning task graph")
 
     def _prune_workers(self):
         remove_workers = []


Test patch:
Repository: spotify/luigi

Issue:
race condition on lock :(
I think there is a race condition on 
**luigi/lock.py", line 93, in acquire_for**

```python
# Check if there is a pid file corresponding to this name
if not os.path.exists(pid_dir):
  os.mkdir(pid_dir)
```
between `if not [...]` and `mkdir` another process could have created the pid_dir resulting  in an unhandled OSError exception.

I think that ignore the exception if the folder exists will be enough, or perhaps just create the folder and say sorry if it was just there...



Discussion:
Feel free to submit a Pull Request if you believe `os.mkdir` should be surrounded in `try/except` (or whatever solution you believe to be best)!

Thanks

Resolving patch:
diff --git a/luigi/lock.py b/luigi/lock.py
--- a/luigi/lock.py
+++ b/luigi/lock.py
@@ -21,6 +21,7 @@
 """
 from __future__ import print_function
 
+import errno
 import hashlib
 import os
 import sys
@@ -102,10 +103,14 @@ def acquire_for(pid_dir, num_available=1, kill_signal=None):
 
     my_pid, my_cmd, pid_file = get_info(pid_dir)
 
-    # Check if there is a pid file corresponding to this name
-    if not os.path.exists(pid_dir):
+    # Create a pid file if it does not exist
+    try:
         os.mkdir(pid_dir)
         os.chmod(pid_dir, 0o777)
+    except OSError as exc:
+        if exc.errno != errno.EEXIST:
+            raise
+        pass
 
     # Let variable "pids" be all pids who exist in the .pid-file who are still
     # about running the same command.


Test patch:
Repository: spotify/luigi

Issue:
Support for Python 3 pathlib Paths in LocalTarget
It would be nice to be able to use `Path` based objects from Python 3's [`pathlib`](https://docs.python.org/3/library/pathlib.html) when defining paths for `LocalTarget`s. Currently, you need to cast them to strings or the target methods like `exists` and `open` won't work.

```
In [9]: lt = luigi.LocalTarget(Path('~') / 'kk')
In [10]: lt.exists()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-11-b13e17b73d35> in <module>()
----> 1 lt.exists()

~/.virtualenvs/bishop/lib/python3.6/site-packages/luigi/target.py in exists(self)
    238         """
    239         path = self.path
--> 240         if '*' in path or '?' in path or '[' in path or '{' in path:
    241             logger.warning("Using wildcards in path %s might lead to processing of an incomplete dataset; "
    242                            "override exists() to suppress the warning.", path)

TypeError: argument of type 'PosixPath' is not iterable
In [11]: lt = luigi.LocalTarget(str(Path('~') / 'kk'))
In [12]: lt.exists()
Out[12]: False

```
  


Discussion:
I'd love to see this too.

Resolving patch:
diff --git a/luigi/target.py b/luigi/target.py
--- a/luigi/target.py
+++ b/luigi/target.py
@@ -199,6 +199,13 @@ class FileSystemTarget(Target):
 
     Methods of FileSystemTarget raise :py:class:`FileSystemException` if there is a problem
     completing the operation.
+
+    Usage:
+        .. code-block:: python
+
+            target = FileSystemTarget('~/some_file.txt')
+            target = FileSystemTarget(pathlib.Path('~') / 'some_file.txt')
+            target.exists()  # False
     """
 
     def __init__(self, path):
@@ -207,7 +214,8 @@ def __init__(self, path):
 
         :param str path: the path associated with this FileSystemTarget.
         """
-        self.path = path
+        # cast to str to allow path to be objects like pathlib.PosixPath and py._path.local.LocalPath
+        self.path = str(path)
 
     @abc.abstractproperty
     def fs(self):


Test patch:
diff --git a/test/local_target_test.py b/test/local_target_test.py
--- a/test/local_target_test.py
+++ b/test/local_target_test.py
@@ -21,6 +21,7 @@
 import os
 import random
 import shutil
+import sys
 from helpers import unittest
 import mock
 
@@ -65,6 +66,18 @@ def test_exists(self):
         p.close()
         self.assertEqual(t.exists(), os.path.exists(self.path))
 
+    @unittest.skipIf(tuple(sys.version_info) < (3, 4), 'only for Python>=3.4')
+    def test_pathlib(self):
+        """Test work with pathlib.Path"""
+        import pathlib
+        path = pathlib.Path(self.path)
+        self.assertFalse(path.exists())
+        target = LocalTarget(path)
+        self.assertFalse(target.exists())
+        with path.open('w') as stream:
+            stream.write('test me')
+        self.assertTrue(target.exists())
+
     def test_gzip_with_module(self):
         t = LocalTarget(self.path, luigi.format.Gzip)
         p = t.open('w')
Repository: spotify/luigi

Issue:
Postgres error if view depends on task_parameters.value
There is a problem related to #2240. Luigi runs `ALTER TABLE task_parameters ALTER COLUMN value TYPE TEXT` every time `luigi.db_task_history.DbTaskHistory` is instantiated.
Normally this would be fine because it will not change anything if the type of `value` is already `TEXT`.
However, if you have a Postgres view that depends on this column, Postgres will refuse to run this instruction with an error message like `DETAIL: rule _RETURN on view taskparameter_view depends on column "value"`.

There doesn't seem a sane way to make Postgres accept this statement, it [seems to be necessary](https://stackoverflow.com/questions/17989355/alter-column-ignoring-dependent-views) to drop the view (and all dependent views), run the `ALTER TABLE` statement and re-create them.

It would be much nicer, if Luigi could check if the migration is actually necessary before executing that statement, similar to the way it checks for *sqlite*:

```PYTHON
elif 'sqlite' in engine.dialect.name:
    # SQLite does not support changing column types. A database file will need
    # to be used to pickup this migration change.
    for i in conn.execute('PRAGMA table_info(task_parameters);').fetchall():
        if i['name'] == 'value' and i['type'] != 'TEXT':
            logger.warning(
                'SQLite can not change column types. Please use a new database '
                'to pickup column type changes.'
            )
```



Discussion:
Your point is valid. Feel free to submit a PR with the necessary changes. I'm happy to review :)

Resolving patch:
diff --git a/luigi/db_task_history.py b/luigi/db_task_history.py
--- a/luigi/db_task_history.py
+++ b/luigi/db_task_history.py
@@ -263,7 +263,9 @@ def _upgrade_schema(engine):
         elif 'mssql' in engine.dialect.name:
             conn.execute('ALTER TABLE task_parameters ALTER COLUMN value TEXT')
         elif 'postgresql' in engine.dialect.name:
-            conn.execute('ALTER TABLE task_parameters ALTER COLUMN value TYPE TEXT')
+            if str([x for x in inspector.get_columns('task_parameters')
+                    if x['name'] == 'value'][0]['type']) != 'TEXT':
+                conn.execute('ALTER TABLE task_parameters ALTER COLUMN value TYPE TEXT')
         elif 'sqlite' in engine.dialect.name:
             # SQLite does not support changing column types. A database file will need
             # to be used to pickup this migration change.


Test patch:
Repository: spotify/luigi

Issue:
HdfsTarget is not workging in luigi > 2.6.1
This code runs correct under luigi <= 2.6.1:

```python
from luigi.contrib import hdfs
import luigi.format

# just to be sure that realy the right version is used...
import pkg_resources
print "luigi ==", pkg_resources.get_distribution("luigi").version
print "snakebite ==", pkg_resources.get_distribution("snakebite").version

destination_file = '/tmp/test/file.gz'

target = hdfs.HdfsTarget(path=destination_file, format=luigi.format.Gzip)

if target.exists():
    target.remove(skip_trash=False)

fsobj = target.open('w')
fsobj.write('lol3\n')
fsobj.close()
```

with luigi 2.6.2 or 2.7.0 it breaks:
```python
luigi == 2.7.0
snakebite == 2.11.0
Traceback (most recent call last):
  File "/opt/tests/hdfs_target.py", line 18, in <module>
    fsobj.close()
  File "/opt/python-2.7.10/lib/python2.7/site-packages/luigi/format.py", line 224, in close
    self._output_pipe.close()
  File "/opt/python-2.7.10/lib/python2.7/site-packages/luigi/contrib/hdfs/format.py", line 51, in close
    remove(self.path)
  File "/opt/python-2.7.10/lib/python2.7/site-packages/luigi/contrib/hdfs/clients.py", line 62, in result
    return getattr(get_autoconfig_client(), method_name)(*args, **kwargs)
  File "/opt/python-2.7.10/lib/python2.7/site-packages/luigi/contrib/hdfs/snakebite_client.py", line 140, in remove
    return list(self.get_bite().delete(self.list_path(path), recurse=recursive))
  File "/opt/python-2.7.10/lib/python2.7/site-packages/snakebite/client.py", line 1540, in wrapped
    yield results.next()
  File "/opt/python-2.7.10/lib/python2.7/site-packages/snakebite/client.py", line 508, in delete
    for item in self._find_items(paths, processor, include_toplevel=True):
  File "/opt/python-2.7.10/lib/python2.7/site-packages/snakebite/client.py", line 1216, in _find_items
    raise FileNotFoundException("`%s': No such file or directory" % path)
snakebite.errors.FileNotFoundException: `/tmp/test/file.gz': No such file or directory
```

client.cfg:
```
[hdfs]
client = snakebite
snakebite_autoconfig = True
tmp_dir: /x/tmp
```

The file is correct in form and content written in "tmp_dir" (/x/tmp/username/test/file.gz-luigitemp-951771388).


Discussion:

Resolving patch:
diff --git a/luigi/contrib/hdfs/format.py b/luigi/contrib/hdfs/format.py
--- a/luigi/contrib/hdfs/format.py
+++ b/luigi/contrib/hdfs/format.py
@@ -49,9 +49,13 @@ def abort(self):
     def close(self):
         super(HdfsAtomicWritePipe, self).close()
         try:
-            remove(self.path)
-        except HDFSCliError:
-            pass
+            if exists(self.path):
+                remove(self.path)
+        except Exception as ex:
+            if isinstance(ex, HDFSCliError) or ex.args[0].contains("FileNotFoundException"):
+                pass
+            else:
+                raise ex
         if not all(result['result'] for result in rename(self.tmppath, self.path) or []):
             raise HdfsAtomicWriteError('Atomic write to {} failed'.format(self.path))
 


Test patch:
Repository: spotify/luigi

Issue:
AttributeError during logging setup with Python 2.7
Hi,

I upgraded to luigi 2.8.0 and experience an `AttributeError` that occurs within the new `setup_logging.py`. Here is the important part of the stack trace:

```
  File ".../lib/python2.7/site-packages/luigi/cmdline.py", line 9, in luigi_run
    run_with_retcodes(argv)
  File ".../lib/python2.7/site-packages/luigi/retcodes.py", line 81, in run_with_retcodes
    InterfaceLogging.setup(env_params)
  File ".../lib/python2.7/site-packages/luigi/setup_logging.py", line 74, in setup
    configured = cls._section(opts)
  File ".../lib/python2.7/site-packages/luigi/setup_logging.py", line 42, in _section
    logging_config = cls.config['logging']
AttributeError: LuigiConfigParser instance has no attribute '__getitem__'
```

`cls.config` is a `ConfigParser` instance which does not implement `__getitem__` in Python 2.7 (which I'm forced to use -.-). This could be fixed by using `cls.config.options('logging')` instead. I checked, and this is the only place where items are used.

https://github.com/spotify/luigi/blob/0a098f6f99da0bad03af56a057b9a15254d1a957/luigi/setup_logging.py#L42

Adding @orsinium here who is maybe faster to fix this. I could create a PR on Friday.


Discussion:
Wow. Thank you. I'll fix it asap today.
Thanks @riga for reporting and @orsinium for quick response! I'll review once it comes through and see about pushing a 2.8.1 hotfix (cc: @honnix )

Resolving patch:
diff --git a/luigi/setup_logging.py b/luigi/setup_logging.py
--- a/luigi/setup_logging.py
+++ b/luigi/setup_logging.py
@@ -22,7 +22,7 @@
 import logging
 import logging.config
 import os.path
-from luigi.configuration import get_config
+from luigi.configuration import get_config, LuigiConfigParser
 
 # In python3 ConfigParser was renamed
 # https://stackoverflow.com/a/41202010
@@ -38,6 +38,8 @@ class BaseLogging(object):
     @classmethod
     def _section(cls, opts):
         """Get logging settings from config file section "logging"."""
+        if isinstance(cls.config, LuigiConfigParser):
+            return False
         try:
             logging_config = cls.config['logging']
         except (TypeError, KeyError, NoSectionError):


Test patch:
diff --git a/test/setup_logging_test.py b/test/setup_logging_test.py
--- a/test/setup_logging_test.py
+++ b/test/setup_logging_test.py
@@ -1,5 +1,5 @@
 from luigi.setup_logging import DaemonLogging, InterfaceLogging
-from luigi.configuration import LuigiTomlParser, get_config
+from luigi.configuration import LuigiTomlParser, LuigiConfigParser, get_config
 from helpers import unittest
 
 
@@ -42,6 +42,11 @@ def test_section(self):
         result = self.cls._section(None)
         self.assertFalse(result)
 
+    def test_section_cfg(self):
+        self.cls.config = LuigiConfigParser.instance()
+        result = self.cls._section(None)
+        self.assertFalse(result)
+
     def test_cfg(self):
         self.cls.config = LuigiTomlParser()
         self.cls.config.data = {}