internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: sql-machine-learning/elasticdl

Issue:
Set version for develop branch.
According to [PEP 440](https://www.python.org/dev/peps/pep-0440/#developmental-releases), we should set version `X.YaN.devM`.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,7 +24,7 @@
 
 setup(
     name="elasticdl",
-    version="develop",
+    version="0.2.0rc3.dev0",
     description="A Kubernetes-native Deep Learning Framework",
     long_description="ElasticDL is a Kubernetes-native deep learning framework"
     " built on top of TensorFlow 2.0 that supports"
diff --git a/setup_client.py b/setup_client.py
--- a/setup_client.py
+++ b/setup_client.py
@@ -18,7 +18,7 @@
 
 setup(
     name="elasticdl_client",
-    version="develop",
+    version="0.2.0rc3.dev0",
     description="The client command line tool for ElasticDL.",
     long_description="ElasticDL Client is the client command line tool for"
     " ElasticDL. Users can use it to submit distributed ElasticDL jobs to"
diff --git a/setup_preprocessing.py b/setup_preprocessing.py
--- a/setup_preprocessing.py
+++ b/setup_preprocessing.py
@@ -22,7 +22,7 @@
 
 setup(
     name="elasticdl_preprocessing",
-    version="develop",
+    version="0.2.0rc3.dev0",
     description="A feature preprocessing library.",
     long_description="This is an extension of the native Keras Preprocessing"
     " Layers and Feature Column API from TensorFlow. We can develop our model"


Test patch:
Repository: sql-machine-learning/elasticdl

Issue:
AllReduce may hang when one worker starts firsts
This is because in Horovod, when `hvd.size() == 1`, it won't use allreduce op.

https://github.com/horovod/horovod/blob/e4554de96100f5a0e8686cd41cf99a6fe8a71e62/horovod/tensorflow/__init__.py#L159


Discussion:

Resolving patch:
diff --git a/elasticdl/python/common/constants.py b/elasticdl/python/common/constants.py
--- a/elasticdl/python/common/constants.py
+++ b/elasticdl/python/common/constants.py
@@ -94,3 +94,4 @@ class HorovodEnv(object):
     CONTROLLER = "HOROVOD_CONTROLLER"
     CPU_OPERATIONS = "HOROVOD_CPU_OPERATIONS"
     HOSTNAME = "HOROVOD_HOSTNAME"
+    ELASTIC = "HOROVOD_ELASTIC"
diff --git a/elasticdl/python/master/k8s_instance_manager.py b/elasticdl/python/master/k8s_instance_manager.py
--- a/elasticdl/python/master/k8s_instance_manager.py
+++ b/elasticdl/python/master/k8s_instance_manager.py
@@ -348,6 +348,10 @@ def _event_cb(self, event):
                 logger.error("Unknown pod name: %s" % pod_name)
                 return
 
+            if self._rendezvous_server:
+                self._worker_addrs = self._get_alive_worker_addr()
+                self._rendezvous_server.set_worker_hosts(self._worker_addrs)
+
         if relaunch_worker and worker_id >= 0:
             logger.info("Relaunching worker.")
             new_worker_id = self._next_worker_id()
@@ -363,10 +367,6 @@ def _event_cb(self, event):
             # tolerance.
             self._start_ps(ps_id)
 
-        if self._rendezvous_server:
-            self._worker_addrs = self._get_alive_worker_addr()
-            self._rendezvous_server.set_worker_hosts(self._worker_addrs)
-
     def get_alive_workers(self):
         alive_workers = []
         for pod_name, _, phase in self._worker_pods_ip_phase.values():
diff --git a/elasticdl/python/worker/allreduce_trainer.py b/elasticdl/python/worker/allreduce_trainer.py
--- a/elasticdl/python/worker/allreduce_trainer.py
+++ b/elasticdl/python/worker/allreduce_trainer.py
@@ -125,8 +125,12 @@ def init_horovod_if_needed(self):
             )
             os.environ[HorovodEnv.RANK] = str(rank_response.rank_id)
             os.environ[HorovodEnv.SIZE] = str(rank_response.world_size)
+            # Not using Horovod elastic feature in init, but need it for
+            # allreduce to call allreduce op when size=1.
+            os.environ[HorovodEnv.ELASTIC] = str(0)
             hvd.shutdown()
             hvd.init()
+            os.environ[HorovodEnv.ELASTIC] = str(1)
             self._world_size = hvd.size()
             self._rendezvous_id = rank_response.rendezvous_id
             self._need_broadcast = True


Test patch:
Repository: sql-machine-learning/elasticdl

Issue:
The gradient indices of embedding is not original inputs
The `IndexedSlice.indices` is not the input ids but the indices of `tf.unique(ids)`

https://github.com/sql-machine-learning/elasticdl/blob/83b37c0c0c001d71cf0438c7b14adf29a163b835/elasticdl/python/worker/ps_trainer.py#L262-L265


Discussion:

Resolving patch:
diff --git a/elasticdl/python/worker/ps_trainer.py b/elasticdl/python/worker/ps_trainer.py
--- a/elasticdl/python/worker/ps_trainer.py
+++ b/elasticdl/python/worker/ps_trainer.py
@@ -262,7 +262,7 @@ def _report_gradient(self, gradients):
                     grad = Tensor(
                         name,
                         edl_embedding_grads[i + bet_number].values.numpy(),
-                        edl_embedding_grads[i + bet_number].indices.numpy(),
+                        embedding_and_ids[i].batch_ids.numpy(),
                     )
                     edl_grads.append(grad)
                 bet_number += len(embedding_and_ids)


Test patch:
Repository: sql-machine-learning/elasticdl

Issue:
Write a tutorial of elastic AllReduce with PyTorch



Discussion:

Resolving patch:
diff --git a/model_zoo/mnist/mnist_pytorch.py b/model_zoo/mnist/mnist_pytorch.py
--- a/model_zoo/mnist/mnist_pytorch.py
+++ b/model_zoo/mnist/mnist_pytorch.py
@@ -11,31 +11,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""
-Download the mnist dataset from
-https://s3.amazonaws.com/fast-ai-imageclas/mnist_png.tgz
-and then untar it into ${data_store_dir}. Using minikube, we can use the
-following command to submit a training job with these codes.
-
-elasticdl train \
-  --image_name=elasticdl:pt_mnist_allreduce  \
-  --job_command="python -m model_zoo.mnist.mnist_pytorch \
-      --training_data=/local_data/mnist_png/training \
-      --validation_data=/local_data/mnist_png/testing" \
-  --num_minibatches_per_task=2 \
-  --num_workers=2 \
-  --worker_pod_priority=0.5 \
-  --master_resource_request="cpu=0.2,memory=1024Mi" \
-  --master_resource_limit="cpu=1,memory=2048Mi" \
-  --worker_resource_request="cpu=0.3,memory=1024Mi" \
-  --worker_resource_limit="cpu=1,memory=2048Mi" \
-  --envs="USE_TORCH=true,HOROVOD_GLOO_TIMEOUT_SECONDS=60,PYTHONUNBUFFERED=0" \
-  --job_name=test-mnist-allreduce \
-  --image_pull_policy=Never \
-  --volume="host_path=${data_store_dir},mount_path=/local_data" \
-  --distribution_strategy=AllreduceStrategy \
-"""
-
 import argparse
 import sys
 
@@ -49,7 +24,8 @@
 from torch.optim.lr_scheduler import StepLR
 from torch.utils.data import DataLoader, Dataset
 
-from elasticai_api.pytorch.controller import create_elastic_controller
+from elasticai_api.common.data_shard_service import RecordIndexService
+from elasticai_api.pytorch.controller import PyTorchAllReduceController
 from elasticai_api.pytorch.optimizer import DistributedOptimizer
 
 
@@ -131,15 +107,14 @@ def train(args):
     train_data = torchvision.datasets.ImageFolder(args.training_data)
     test_data = torchvision.datasets.ImageFolder(args.validation_data)
 
-    allreduce_controller = create_elastic_controller(
+    data_shard_service = RecordIndexService(
         batch_size=args.batch_size,
         dataset_size=len(train_data.imgs),
         num_epochs=args.num_epochs,
         shuffle=True,
+        dataset_name="mnist_training_data",
     )
-    train_dataset = ElasticDataset(
-        train_data.imgs, allreduce_controller.data_shard_service
-    )
+    train_dataset = ElasticDataset(train_data.imgs, data_shard_service)
     train_loader = DataLoader(
         dataset=train_dataset, batch_size=args.batch_size, num_workers=2
     )
@@ -155,6 +130,7 @@ def train(args):
     scheduler = StepLR(optimizer, step_size=1, gamma=0.5)
 
     # Set the model and optimizer to broadcast.
+    allreduce_controller = PyTorchAllReduceController(data_shard_service)
     allreduce_controller.set_broadcast_model(model)
     allreduce_controller.set_broadcast_optimizer(optimizer)
     epoch = 0


Test patch:
Repository: aio-libs/aiohttp

Issue:
Fix proxy issue
Code to reproduce:

```
import aiohttp
import asyncio


@asyncio.coroutine
def go():
    connector = aiohttp.ProxyConnector('http://37.44.192.55:8888')
    resp = yield from aiohttp.request(
        'get',
        'http://httpbin.org/cookies/set?k1=v1&k2=v2',
        connector=connector)
    body = yield from resp.read_and_close()
    print(body)


asyncio.get_event_loop().run_until_complete(go())
```



Discussion:
I will look into


Resolving patch:
diff --git a/aiohttp/client.py b/aiohttp/client.py
--- a/aiohttp/client.py
+++ b/aiohttp/client.py
@@ -153,7 +153,7 @@ def request(method, url, *,
 
             url = urllib.parse.urldefrag(r_url)[0]
             if url:
-                resp.close()
+                yield from asyncio.Task(resp.read_and_close(), loop=loop)
                 continue
 
         break


Test patch:
Repository: aio-libs/aiohttp

Issue:
body for HTTP GET request
When aiohttp.request gets `GET` method with specified `data` that merges data to url along with `params`.

I think that's wrong. HTTP protocol standard (http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) says nothing about presence/absence body for GET methods.

Moreover elasticsearch uses GETs with body for some REST API methods: see http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-multi-get.html and http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-multi-search.html#search-multi-search

I guess to always process `data` as for `POST` request if specified.



Discussion:
http://trac.tools.ietf.org/wg/httpbis/trac/ticket/19
and
https://groups.yahoo.com/neo/groups/rest-discuss/conversations/messages/9962

few more interesting discussions:
http://lists.w3.org/Archives/Public/ietf-http-wg/2002JulSep/0031.html
http://lists.w3.org/Archives/Public/ietf-http-wg/2007JanMar/0076.html

TL;DR technically, it's ok to send body with GET/DELETE requests. Semantically - it's not. In real world only few uses such "feature".

+1

 let's  unify 'data' processing for all type of requests


Resolving patch:
diff --git a/aiohttp/client.py b/aiohttp/client.py
--- a/aiohttp/client.py
+++ b/aiohttp/client.py
@@ -189,7 +189,7 @@ def __init__(self, method, url, *,
 
         self.update_version(version)
         self.update_host(url)
-        self.update_path(params, data)
+        self.update_path(params)
         self.update_headers(headers)
         self.update_cookies(cookies)
         self.update_content_encoding()
@@ -264,7 +264,7 @@ def update_version(self, version):
                     .format(version)) from None
         self.version = version
 
-    def update_path(self, params, data):
+    def update_path(self, params):
         """Build path."""
         # extract path
         scheme, netloc, path, query, fragment = urllib.parse.urlsplit(self.url)
@@ -276,12 +276,6 @@ def update_path(self, params, data):
         elif isinstance(params, MultiDict):
             params = list(params.items(getall=True))
 
-        # for GET request include data to query params
-        if data and self.method in self.GET_METHODS:
-            if isinstance(data, dict):
-                data = data.items()
-            params = list(itertools.chain(params or (), data))
-
         if params:
             params = urllib.parse.urlencode(params)
             if query:


Test patch:
diff --git a/tests/test_client.py b/tests/test_client.py
--- a/tests/test_client.py
+++ b/tests/test_client.py
@@ -508,7 +508,8 @@ def test_get_with_data(self):
         for meth in ClientRequest.GET_METHODS:
             req = ClientRequest(
                 meth, 'http://python.org/', data={'life': '42'})
-            self.assertEqual('/?life=42', req.path)
+            self.assertEqual('/', req.path)
+            self.assertEqual(b'life=42', req.body)
 
     def test_bytes_data(self):
         for meth in ClientRequest.POST_METHODS:
Repository: aio-libs/aiohttp

Issue:
HttpMessage._add_default_headers should not overwrite existing headers
Simply using setdefault should suffice rather than extend.

This is crucial for custom protocol extensions where the Server / User-Agent headers is used to
notify clients/servers about capabilities.



Discussion:

Resolving patch:
diff --git a/aiohttp/protocol.py b/aiohttp/protocol.py
--- a/aiohttp/protocol.py
+++ b/aiohttp/protocol.py
@@ -521,8 +521,6 @@ class HttpMessage:
     # this is useful for wsgi's start_response implementation.
     _send_headers = False
 
-    _has_user_agent = False
-
     def __init__(self, transport, version, close):
         self.transport = transport
         self.version = version
@@ -595,9 +593,6 @@ def add_header(self, name, value):
             self.chunked = value.lower().strip() == 'chunked'
 
         elif name not in self.HOP_HEADERS:
-            if name == 'USER-AGENT':
-                self._has_user_agent = True
-
             # ignore hop-by-hop headers
             self.headers.add(name, value)
 
@@ -805,8 +800,6 @@ class Response(HttpMessage):
         'TRAILERS',
         'TRANSFER-ENCODING',
         'UPGRADE',
-        'SERVER',
-        'DATE',
     }
 
     @staticmethod
@@ -833,8 +826,10 @@ def __init__(self, transport, status,
     def _add_default_headers(self):
         super()._add_default_headers()
 
-        self.headers.extend((('DATE', format_date_time(None)),
-                             ('SERVER', self.SERVER_SOFTWARE),))
+        if 'DATE' not in self.headers:
+            # format_date_time(None) is quite expensive
+            self.headers.setdefault('DATE', format_date_time(None))
+        self.headers.setdefault('SERVER', self.SERVER_SOFTWARE)
 
 
 class Request(HttpMessage):
@@ -853,5 +848,4 @@ def __init__(self, transport, method, path,
     def _add_default_headers(self):
         super()._add_default_headers()
 
-        if not self._has_user_agent:
-            self.headers['USER-AGENT'] = self.SERVER_SOFTWARE
+        self.headers.setdefault('USER-AGENT', self.SERVER_SOFTWARE)


Test patch:
diff --git a/tests/test_http_protocol.py b/tests/test_http_protocol.py
--- a/tests/test_http_protocol.py
+++ b/tests/test_http_protocol.py
@@ -479,3 +479,18 @@ def test_write_drain(self):
 
         res = msg.write(b'1')
         self.assertEqual(res, ())
+
+    def test_dont_override_request_headers_with_default_values(self):
+        msg = protocol.Request(
+            self.transport, 'GET', '/index.html', close=True)
+        msg.add_header('USER-AGENT', 'custom')
+        msg._add_default_headers()
+        self.assertEqual('custom', msg.headers['USER-AGENT'])
+
+    def test_dont_override_response_headers_with_default_values(self):
+        msg = protocol.Response(self.transport, 200, http_version=(1, 0))
+        msg.add_header('DATE', 'now')
+        msg.add_header('SERVER', 'custom')
+        msg._add_default_headers()
+        self.assertEqual('custom', msg.headers['SERVER'])
+        self.assertEqual('now', msg.headers['DATE'])
Repository: aio-libs/aiohttp

Issue:
Connection not closed when request is cancelled
When making a lot of concurrent client requests to the same host using connection pooling, TCPConnector will create a lot of connections. When cancelling these client requests, not all connections are closed.

This is not easy to reproduce. I have a long running server process, and I can see the open sockets increasing over time.

Here's a gist that reproduces the conditions. When the code prints DONE, leave the process running and look at the open sockets (e.g. lsof -p <PID> | grep http). After the keepalive_timeout of 30 seconds, most connections are closed, but occasionally (1 in 5 runs for me), there is a dangling socket. You may need to change the sleep timeout for cancellation depending on how fast your connection is:
https://gist.github.com/nbraem/0e5178288ffd94372062

I have a fix, I'll send a pull request shortly.



Discussion:

Resolving patch:
diff --git a/aiohttp/connector.py b/aiohttp/connector.py
--- a/aiohttp/connector.py
+++ b/aiohttp/connector.py
@@ -199,7 +199,9 @@ def _release(self, key, req, transport, protocol, *, should_close=False):
 
         reader = protocol.reader
         if should_close or (reader.output and not reader.output.at_eof()):
-            self._conns.pop(key, None)
+            conns = self._conns.get(key)
+            if conns is not None and len(conns) == 0:
+                self._conns.pop(key, None)
             transport.close()
         else:
             conns = self._conns.get(key)


Test patch:
Repository: aio-libs/aiohttp

Issue:
Optionally use cchardet if installed
see https://github.com/PyYoshi/cChardet



Discussion:
+1


Resolving patch:
diff --git a/aiohttp/client_reqrep.py b/aiohttp/client_reqrep.py
--- a/aiohttp/client_reqrep.py
+++ b/aiohttp/client_reqrep.py
@@ -9,7 +9,10 @@
 import urllib.parse
 import warnings
 
-import chardet
+try:
+    import cchardet as chardet
+except ImportError:
+    import chardet
 
 import aiohttp
 from . import hdrs, helpers, streams


Test patch:
Repository: aio-libs/aiohttp

Issue:
Is break in finally intentional?
In low-level server code I see `break` statements inside `finally` blocks like https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/server.py#L290

It looks suspicious: `break` inside `finally` block discards saved exception (see https://docs.python.org/3.5/reference/simple_stmts.html#break). 
I believe `return` should be used instead, it does the same in the case (stop reading incoming data) but it is show the intention much more explicitly.

@fafhrd91 you are author of these lines. Do you remember the motivation for `break` statement?



Discussion:
I do not really remember, but good catch let's replace with returns 

Sent from my iPhone

> On Jul 27, 2015, at 7:53 AM, Andrew Svetlov notifications@github.com wrote:
> 
> In low-level server code I see break statements inside finally blocks like https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/server.py#L290
> 
> It looks suspicious: break inside finally block discards saved exception (see https://docs.python.org/3.5/reference/simple_stmts.html#break). 
> I believe return should be used instead, it does the same in the case (stop reading incoming data) but it is shows the intention much more explicitly.
> 
> @fafhrd91 you are author of these lines. Do you remember the motivation for break statement?
> 
> —
> Reply to this email directly or view it on GitHub.


Resolving patch:
diff --git a/aiohttp/server.py b/aiohttp/server.py
--- a/aiohttp/server.py
+++ b/aiohttp/server.py
@@ -275,7 +275,7 @@ def start(self):
                 if self.debug:
                     self.log_exception(
                         'Ignored premature client disconnection.')
-                break
+                return
             except errors.HttpProcessingError as exc:
                 if self.transport is not None:
                     yield from self.handle_error(exc.code, message,
@@ -287,13 +287,13 @@ def start(self):
             finally:
                 if self.transport is None:
                     self.log_debug('Ignored premature client disconnection.')
-                    break
+                    return
 
                 if payload and not payload.is_eof():
                     self.log_debug('Uncompleted request.')
                     self._request_handler = None
                     self.transport.close()
-                    break
+                    return
                 else:
                     reader.unset_parser()
 
@@ -311,10 +311,10 @@ def start(self):
                         self.log_debug('Close client connection.')
                         self._request_handler = None
                         self.transport.close()
-                        break
+                        return
                 else:
                     # connection is closed
-                    break
+                    return
 
     def handle_error(self, status=500,
                      message=None, payload=None, exc=None, headers=None):


Test patch:
Repository: aio-libs/aiohttp

Issue:
Incorrect remote_addr in access log
All records contain 127.0.0.1 as remote_addr



Discussion:

Resolving patch:
diff --git a/aiohttp/helpers.py b/aiohttp/helpers.py
--- a/aiohttp/helpers.py
+++ b/aiohttp/helpers.py
@@ -239,7 +239,7 @@ def atoms(message, environ, response, transport, request_time):
 
     if transport is not None:
         remote_addr = parse_remote_addr(
-            transport.get_extra_info('addr', '127.0.0.1'))
+            transport.get_extra_info('peername', ('127.0.0.1', )))
     else:
         remote_addr = ('',)
 


Test patch:
Repository: aio-libs/aiohttp

Issue:
aiohttp filtering out "Authorization" header
Apparently aiohttp is filtering out the "Authorization" header in aiohttp.wsgi:69 in create_wsgi_environ.

This bug was found while using aiopyramid + jwtauth, you can find more details (and an example project) on https://github.com/housleyjk/aiopyramid/issues/14



Discussion:
It seems my bad. According to wsgi and cgi specs, AUTHORIZATION header should be removed, but.

In our case, server does no handle any basic auth stuff, so this headers may only be handled at application level, so aiohttp should not remove it.


Resolving patch:
diff --git a/aiohttp/wsgi.py b/aiohttp/wsgi.py
--- a/aiohttp/wsgi.py
+++ b/aiohttp/wsgi.py
@@ -66,9 +66,7 @@ def create_wsgi_environ(self, message, payload):
         script_name = self.SCRIPT_NAME
 
         for hdr_name, hdr_value in message.headers.items():
-            if hdr_name == 'AUTHORIZATION':
-                continue
-            elif hdr_name == 'SCRIPT_NAME':
+            if hdr_name == 'SCRIPT_NAME':
                 script_name = hdr_value
             elif hdr_name == 'CONTENT-TYPE':
                 environ['CONTENT_TYPE'] = hdr_value


Test patch:
diff --git a/tests/test_wsgi.py b/tests/test_wsgi.py
--- a/tests/test_wsgi.py
+++ b/tests/test_wsgi.py
@@ -258,14 +258,15 @@ def test_dont_unquote_environ_path_info(self):
         environ = self._make_one()
         self.assertEqual(environ['PATH_INFO'], path)
 
-    def test_not_add_authorization(self):
-        self.headers.extend({'AUTHORIZATION': 'spam',
-                             'X-CUSTOM-HEADER': 'eggs'})
+    def test_authorization(self):
+        # This header should be removed according to CGI/1.1 and WSGI but
+        # in our case basic auth is not handled by server, so should
+        # not be removed
+        self.headers.extend({'AUTHORIZATION': 'spam'})
         self.message = protocol.RawRequestMessage(
             'GET', '/', (1, 1), self.headers, True, 'deflate')
         environ = self._make_one()
-        self.assertEqual('eggs', environ['HTTP_X_CUSTOM_HEADER'])
-        self.assertFalse('AUTHORIZATION' in environ)
+        self.assertEqual('spam', environ['HTTP_AUTHORIZATION'])
 
     def test_http_1_0_no_host(self):
         headers = multidict.MultiDict({})