internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mars-project/mars

Issue:
[BUG] Intensive embarrassing parallel jobs may cause execution timeout
**Describe the bug**
When there are a number of long-time tasks submitted to workers, job assignments may take long time to respond, which prone to timeouts. We need to adjust the timeout strategy given cluster loads / executing tasks.



Discussion:

Resolving patch:
diff --git a/mars/scheduler/assigner.py b/mars/scheduler/assigner.py
--- a/mars/scheduler/assigner.py
+++ b/mars/scheduler/assigner.py
@@ -406,7 +406,12 @@ def _allocate_resource(self, session_id, op_key, op_info, target_worker=None, re
                 rejects.append(worker_ep)
 
         if timeout_on_fail:
-            raise TimeoutError(f'Assign resources to operand {op_key} timed out')
+            running_ops = sum(len(metrics.get('progress', dict()).get(str(session_id), dict()))
+                              for metrics in self._worker_metrics.values())
+            if running_ops == 0:
+                raise TimeoutError(f'Assign resources to operand {op_key} timed out')
+            else:
+                self._session_last_assigns[session_id] = time.time()
         return None, rejects
 
     def _get_chunks_meta(self, session_id, keys):
diff --git a/mars/scheduler/graph.py b/mars/scheduler/graph.py
--- a/mars/scheduler/graph.py
+++ b/mars/scheduler/graph.py
@@ -735,6 +735,11 @@ def _assign_initial_workers(self, analyzer):
         ext_chunks_to_inputs = analyzer.collect_external_input_chunks(initial=True)
         input_chunk_metas = self._collect_external_input_metas(ext_chunks_to_inputs)
 
+        invalid_keys = [k for k, v in input_chunk_metas.items() if v is None]
+        if invalid_keys:
+            raise KeyError(f'Input metas for data chunks {invalid_keys!r} missing when '
+                           f'executing graph {self._graph_key}')
+
         def _do_assign():
             # do placements
             return self.assign_operand_workers(


Test patch:
Repository: mars-project/mars

Issue:
[BUG] mars.utils._get_ports_from_netstat hang
<!--
Thank you for your contribution!

Please review https://github.com/mars-project/mars/blob/master/CONTRIBUTING.rst before opening an issue.
-->

**Describe the bug**
`mars.utils._get_ports_from_netstat` hangs on `subprocess.Popen.wait`. It's documented in [Popen.wait](https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait)

> Popen.wait():
> 
> Wait for child process to terminate. Set and return returncode attribute.
> 
> Warning: This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.
> 

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version:python 3.8
2. The version of Mars you use: mars master
3. Versions of crucial packages, such as numpy, scipy and pandas: latest
4. Full stack of the error.
![image](https://user-images.githubusercontent.com/12445254/116360188-3f5fb480-a832-11eb-85c5-5bb45604d1d4.png)
5. Minimized code to reproduce the error.
Call `mars.utils._get_ports_from_netstat` when there're plenty of unused ports




Discussion:

Resolving patch:
diff --git a/mars/utils.py b/mars/utils.py
--- a/mars/utils.py
+++ b/mars/utils.py
@@ -204,27 +204,25 @@ def _get_ports_from_netstat():
     while True:
         p = subprocess.Popen('netstat -a -n -p tcp'.split(), stdout=subprocess.PIPE)
         try:
-            p.wait(5)
-            break
+            outs, _ = p.communicate(timeout=5)
+            outs = to_str(outs).split('\n')
+            occupied = set()
+            for line in outs:
+                if '.' not in line:
+                    continue
+                for part in line.split():
+                    if '.' in part:
+                        _, port_str = part.rsplit('.', 1)
+                        if port_str == '*':
+                            continue
+                        port = int(port_str)
+                        if LOW_PORT_BOUND <= port <= HIGH_PORT_BOUND:
+                            occupied.add(int(port_str))
+                        break
+            return occupied
         except subprocess.TimeoutExpired:
-            p.terminate()
-            continue
-    occupied = set()
-    for line in p.stdout:
-        line = to_str(line)
-        if '.' not in line:
+            p.kill()
             continue
-        for part in line.split():
-            if '.' in part:
-                _, port_str = part.rsplit('.', 1)
-                if port_str == '*':
-                    continue
-                port = int(port_str)
-                if LOW_PORT_BOUND <= port <= HIGH_PORT_BOUND:
-                    occupied.add(int(port_str))
-                break
-    p.stdout.close()
-    return occupied
 
 
 def get_next_port(typ=None, occupy=True):


Test patch:
Repository: mars-project/mars

Issue:
Add docs to explain the internals
<!--
Thank you for your contribution!

Please review https://github.com/mars-project/mars/blob/master/CONTRIBUTING.rst before opening an issue.
-->

**Is your feature request related to a problem? Please describe.**

We should add some docs to explain the internals of Mars.


Discussion:
Contents

1. Structure
   1. Overview
   2. Code structure
   3. How tasks are executed
2. Oscar actor system
   1. Overview
   2. Actor creation
   3. Actor execution
   4. Communication and routing
   5. Batch methods
3. Service topics
   1. Cluster
   2. Lifecycle
   3. Meta
   4. Scheduling
   5. Session
   6. Storage
   7. Subtask
   8. Task
   9. Web

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -196,7 +196,6 @@
 
 
 import sphinx
-from sphinx.util import rpartition
 from sphinx.ext.autodoc import MethodDocumenter, Documenter, AttributeDocumenter
 
 
@@ -250,8 +249,8 @@ def resolve_name(self, modname, parents, path, base):
             # HACK: this is added in comparison to ClassLevelDocumenter
             # mod_cls still exists of class.accessor, so an extra
             # rpartition is needed
-            modname, accessor = rpartition(mod_cls, ".")
-            modname, cls = rpartition(modname, ".")
+            modname, _, accessor = mod_cls.rpartition(".")
+            modname, _, cls = modname.rpartition(".")
             if cls:
                 parents = [cls, accessor]
             else:
diff --git a/mars/services/scheduling/supervisor/queueing.py b/mars/services/scheduling/supervisor/queueing.py
--- a/mars/services/scheduling/supervisor/queueing.py
+++ b/mars/services/scheduling/supervisor/queueing.py
@@ -176,8 +176,6 @@ async def submit_subtasks(self, band: Tuple = None, limit: Optional[int] = None)
             for stid in non_submitted_ids:
                 heapq.heappush(task_queue, submit_items[stid])
 
-        if limit is None:
-            logger.warning('Submitted %d tasks without limits', len(submit_aio_tasks))
         if submit_aio_tasks:
             yield asyncio.gather(*submit_aio_tasks)
 


Test patch:
Repository: mars-project/mars

Issue:
Tileable Progress can be illustrated on Task Detail DAG
**Is your feature request related to a problem? Please describe.**
On the Task Detail DAG, we can paint the color of each tileable based on its progress. This will help us to oversee the status of the task. 

Right now, all the tileables on the DAG are filled with a yellow color:
![Picture1](https://user-images.githubusercontent.com/69881587/129529255-1dde9071-e7ee-490f-9583-8abd1c8423ca.png)

 but if we paint the color based on the percentage finished, the graph will be more informative. A tileable that has been executed can have a filled color, while a tileable that is 70% executed can have 70% of it filled with yellow and the other 30% remain white.  This gives us a more straightforward understanding of the progress of the task overall. 

**Describe the solution you'd like**
We can create another API endpoint that takes a task id and returns a list/dict that contains all tileables in the DAG and their percentage finished as a float between 0 and 1, where 1 means this tileable has been executed and 0 means this tileable has not been executed, and a float in between means how much of this tileable has been executed.

A sample JSON returned may look like this: 

`
{
    [
        { tileable_id_1: 1 },
        { tileable_id_2: 0.7 },
        ...
    ]
}
`

Then on the frontend, we can change the color for tileables based on the corresponded float value.


Discussion:
I'm now considering whether we need to make this API more informative. For instance, we also need to display status information on the DAG. Hence the result of API may like

```json
{
  "tileable1": {
    "progress": 0.7,
    "status": "running",
  },
  "tileable2": {
    "progress": 1.0,
    "status": "finished",
  },
  "tileable3": {
    "status": "cancelled",
  },
  "tileable3": {
    "status": "errored",
  },
  ...
}
```

Resolving patch:
diff --git a/mars/services/task/supervisor/processor.py b/mars/services/task/supervisor/processor.py
--- a/mars/services/task/supervisor/processor.py
+++ b/mars/services/task/supervisor/processor.py
@@ -512,7 +512,13 @@ def get_tileable_graph_as_dict(self):
         node_list = []
         edge_list = []
 
+        visited = set()
+
         for chunk in tileable_graph:
+            if chunk.key in visited:
+                continue
+            visited.add(chunk.key)
+
             node_name = str(chunk.op)
 
             node_list.append({
@@ -527,6 +533,7 @@ def get_tileable_graph_as_dict(self):
                     'toTileableId': chunk.key,
                     'linkType': 1 if is_pure_dep else 0,
                 })
+
         graph_dict = {
             'tileables': node_list,
             'dependencies': edge_list


Test patch:
Repository: mars-project/mars

Issue:
[BUG] The error message is not be output to supervisor's log
<!--
Thank you for your contribution!

Please review https://github.com/mars-project/mars/blob/master/CONTRIBUTING.rst before opening an issue.
-->

**Describe the bug**
For now, the error message raised by tile functions is not printed to supervisor's logs.



Discussion:

Resolving patch:
diff --git a/mars/services/task/supervisor/processor.py b/mars/services/task/supervisor/processor.py
--- a/mars/services/task/supervisor/processor.py
+++ b/mars/services/task/supervisor/processor.py
@@ -14,6 +14,7 @@
 
 import asyncio
 import itertools
+import logging
 import operator
 import sys
 import time
@@ -39,8 +40,10 @@
 from .preprocessor import TaskPreprocessor
 from .stage import TaskStageProcessor
 
+logger = logging.getLogger(__name__)
 
-def _record_error(func: Union[Callable, Coroutine]):
+
+def _record_error(func: Union[Callable, Coroutine]=None, log_when_error=True):
     assert asyncio.iscoroutinefunction(func)
 
     @wraps(func)
@@ -48,6 +51,8 @@ async def inner(processor: "TaskProcessor", *args, **kwargs):
         try:
             return await func(processor, *args, **kwargs)
         except:  # noqa: E722  # nosec  # pylint: disable=bare-except  # pragma: no cover
+            if log_when_error:
+                logger.exception('Unexpected error happens in %s', func)
             processor._err_infos.append(sys.exc_info())
             raise
 


Test patch:
Repository: mars-project/mars

Issue:
[BUG] `test_manhattan_distances_execution` failed for scikit-learn 1.0
<!--
Thank you for your contribution!

Please review https://github.com/mars-project/mars/blob/master/CONTRIBUTING.rst before opening an issue.
-->

**Describe the bug**
For scikit-learn 1.0, `test_manhattan_distances_execution` failed with error as below:
``` Python
    @pytest.mark.parametrize('x, y, is_sparse',
                             [(x1, y1, False),
                              (x2, y2, False),
                              (x3, y3, True),
                              (x4, y4, True)])
    def test_manhattan_distances_execution(setup, x, y, is_sparse):
        if is_sparse:
            rx, ry = raw_sparse_x, raw_sparse_y
        else:
            rx, ry = raw_x, raw_y
    
        sv = [True, False] if not is_sparse else [True]
    
        for sum_over_features in sv:
            d = manhattan_distances(x, y, sum_over_features)
    
            result = d.execute().fetch()
>           expected = sk_manhattan_distances(rx, ry, sum_over_features)
E           TypeError: manhattan_distances() takes from 1 to 2 positional arguments but 3 were given

mars/learn/metrics/pairwise/tests/test_manhattan_distances.py:76: TypeError
```



Discussion:

Resolving patch:
diff --git a/mars/lib/sparse/array.py b/mars/lib/sparse/array.py
--- a/mars/lib/sparse/array.py
+++ b/mars/lib/sparse/array.py
@@ -361,6 +361,16 @@ def __rmul__(self, other):
             return SparseNDArray(x, shape=self.shape)
         return get_array_module(x).asarray(x)
 
+    def __matmul__(self, other):
+        from . import matmul
+
+        return matmul(self, other)
+
+    def __rmatmul__(self, other):
+        from . import matmul
+
+        return matmul(other, self)
+
     def __div__(self, other):
         return self.__truediv__(other)
 


Test patch:
diff --git a/mars/learn/metrics/pairwise/tests/test_manhattan_distances.py b/mars/learn/metrics/pairwise/tests/test_manhattan_distances.py
--- a/mars/learn/metrics/pairwise/tests/test_manhattan_distances.py
+++ b/mars/learn/metrics/pairwise/tests/test_manhattan_distances.py
@@ -73,7 +73,7 @@ def test_manhattan_distances_execution(setup, x, y, is_sparse):
         d = manhattan_distances(x, y, sum_over_features)
 
         result = d.execute().fetch()
-        expected = sk_manhattan_distances(rx, ry, sum_over_features)
+        expected = sk_manhattan_distances(rx, ry, sum_over_features=sum_over_features)
 
         np.testing.assert_almost_equal(result, expected)
 
diff --git a/mars/lib/sparse/tests/test_sparse.py b/mars/lib/sparse/tests/test_sparse.py
--- a/mars/lib/sparse/tests/test_sparse.py
+++ b/mars/lib/sparse/tests/test_sparse.py
@@ -263,6 +263,8 @@ def test_sparse_dot():
     assertArrayEqual(s1.dot(d1), s1.dot(d1))
     assertArrayEqual(d1.dot(s1.T), d1.dot(s1.T.toarray()))
 
+    assertArrayEqual(s1 @ s2.T, s1_data @ s2_data.T)
+
     assertArrayEqual(mls.tensordot(s1, s2.T, axes=(1, 0)), s1.dot(s2.T))
     assertArrayEqual(mls.tensordot(s1, d1, axes=(1, -1)), s1.dot(d1))
     assertArrayEqual(mls.tensordot(d1, s1.T, axes=(0, 0)), d1.dot(s1.T.toarray()))
Repository: mars-project/mars

Issue:
Series.count() returns a series instead of scalar
<!--
Thank you for your contribution!

Please review https://github.com/mars-project/mars/blob/master/CONTRIBUTING.rst before opening an issue.
-->

**Describe the bug**

Series.count() return a series instead of scalar

```
In [13]: raw_data = md.DataFrame([], columns=['id', 'target1', 'target2', 'target3'])                                                                                                                       

In [15]: raw_data['id'].count().execute()                                                                                                                                                                   
Out[15]: Series([], Name: id, dtype: object)
```



Discussion:

Resolving patch:
diff --git a/mars/dataframe/reduction/aggregation.py b/mars/dataframe/reduction/aggregation.py
--- a/mars/dataframe/reduction/aggregation.py
+++ b/mars/dataframe/reduction/aggregation.py
@@ -732,10 +732,14 @@ def execute(cls, ctx, op: "DataFrameAggregate"):
             else:
                 xp = cp if op.gpu else np
                 in_obj = op.inputs[0]
+                in_data = ctx[in_obj.key]
                 if isinstance(in_obj, INDEX_CHUNK_TYPE):
-                    result = op.func[0](ctx[in_obj.key])
+                    result = op.func[0](in_data)
+                elif op.output_types[0] == OutputType.scalar \
+                        and in_data.shape == (0,) and callable(op.func[0]):
+                    result = op.func[0](in_data)
                 else:
-                    result = ctx[in_obj.key].agg(op.raw_func, axis=op.axis)
+                    result = in_data.agg(op.raw_func, axis=op.axis)
 
                 if op.output_types[0] == OutputType.tensor:
                     result = xp.array(result)


Test patch:
diff --git a/mars/dataframe/reduction/tests/test_reduction_execution.py b/mars/dataframe/reduction/tests/test_reduction_execution.py
--- a/mars/dataframe/reduction/tests/test_reduction_execution.py
+++ b/mars/dataframe/reduction/tests/test_reduction_execution.py
@@ -101,6 +101,11 @@ def compute(data, **kwargs):
         reduction_df5 = compute(md.Series(data, chunk_size=3), min_count=21)
         assert np.isnan(reduction_df5.execute().fetch())
 
+    # test reduction on empty series
+    data = pd.Series([], dtype=float, name='a')
+    r = compute(md.Series(data))
+    np.testing.assert_equal(r.execute().fetch(), compute(data))
+
 
 @pytest.mark.parametrize('func_name,func_opts', reduction_functions)
 def test_series_level_reduction(setup, func_name, func_opts: FunctionOptions):
Repository: mars-project/mars

Issue:
[BUG]Cant set queue when start new cluster
When I start a new cluster,I can.t set the queue config.  I read the code and find that, in the `mars.deploy.yarn.client.py`, there is no parameter such as queue in `app_config = MarsApplicationConfig(app_name, supervisor_config=supervisor_config, worker_config=worker_config` 


Discussion:

Resolving patch:
diff --git a/mars/deploy/yarn/client.py b/mars/deploy/yarn/client.py
--- a/mars/deploy/yarn/client.py
+++ b/mars/deploy/yarn/client.py
@@ -83,6 +83,7 @@ def new_cluster(
     log_config=None,
     skein_client=None,
     app_name=None,
+    app_queue=None,
     **kwargs,
 ):
     import skein
@@ -149,7 +150,10 @@ def _override_envs(src, updates):
         cmd_tmpl=cmd_tmpl,
     )
     app_config = MarsApplicationConfig(
-        app_name, supervisor_config=supervisor_config, worker_config=worker_config
+        app_name,
+        app_queue,
+        supervisor_config=supervisor_config,
+        worker_config=worker_config,
     )
 
     skein_client = skein_client or skein.Client()


Test patch:
Repository: mars-project/mars

Issue:
When using results from `series.apply`, unnecessary `reindex` produced
**Is your feature request related to a problem? Please describe.**
When using results from `series.apply`, unnecessary `reindex` produced, which is expensive.

```python
import mars.dataframe as md
df = md.DataFrame([[3, 4, 5], [7, 8, 6], [3, 1, 5]], columns=['a', 'b', 'c'], chunk_size=2)
df['d'] = df.b.apply(lambda x: x + 1)
df.build_graph().to_dot()
```

![image](https://user-images.githubusercontent.com/8284922/139212362-dc648f71-2309-404e-b667-c2fe0f384059.png)

**Describe the solution you'd like**
When index is kept unchanged, reindex is not necessary.


Discussion:

Resolving patch:
diff --git a/mars/dataframe/base/apply.py b/mars/dataframe/base/apply.py
--- a/mars/dataframe/base/apply.py
+++ b/mars/dataframe/base/apply.py
@@ -360,8 +360,8 @@ def _call_series(self, series, dtypes=None, dtype=None, name=None, index=None):
         # for backward compatibility
         dtype = dtype if dtype is not None else dtypes
         if self._convert_dtype:
+            test_series = build_series(series, size=2, name=series.name)
             try:
-                test_series = build_series(series, size=2, name=series.name)
                 with np.errstate(all="ignore"), quiet_stdio():
                     infer_series = test_series.apply(
                         self._func, args=self.args, **self.kwds
@@ -374,7 +374,10 @@ def _call_series(self, series, dtypes=None, dtype=None, name=None, index=None):
             if index is not None:
                 index_value = parse_index(index)
             elif infer_series is not None:
-                index_value = parse_index(infer_series.index)
+                if infer_series.index is test_series.index:
+                    index_value = series.index_value
+                else:  # pragma: no cover
+                    index_value = parse_index(infer_series.index)
             else:
                 index_value = parse_index(None, series)
 


Test patch:
diff --git a/mars/dataframe/base/tests/test_base.py b/mars/dataframe/base/tests/test_base.py
--- a/mars/dataframe/base/tests/test_base.py
+++ b/mars/dataframe/base/tests/test_base.py
@@ -338,6 +338,7 @@ def test_series_apply():
     r = tile(series.apply(np.sqrt))
     assert np.dtype("float64") == r.dtype
     assert r.shape == series.shape
+    assert r.index_value is series.index_value
     assert r.op._op_type_ == opcodes.APPLY
     assert r.op.output_types[0] == OutputType.series
     assert r.chunks[0].shape == (5,)
Repository: mars-project/mars

Issue:
[BUG] DataFrame.merge yields wrong result for one-chunk style merge
<!--
Thank you for your contribution!

Please review https://github.com/mars-project/mars/blob/master/CONTRIBUTING.rst before opening an issue.
-->

**Describe the bug**

The following code:

```python
import pandas as pd
from mars import dataframe as md

x = pd.DataFrame({'id': [1,2,3,4], 'value': [3,4,5,6]})
y = pd.DataFrame({'id': [1,2]})
r = y.merge(x, left_on='id', right_on='id', how='left')
print('pandas = ', r)

xx = md.DataFrame(x, chunk_size=(1,2))
yy1 = md.DataFrame(y, chunk_size=1)
yy2 = md.DataFrame(y, chunk_size=2)

r1 = yy1.merge(xx, left_on='id', right_on='id', how='left').execute()
r2 = yy2.merge(xx, left_on='id', right_on='id', how='left').execute()

print('two chunk = ', r1)
print('one chunk = ', r2)
```

yields

```
pandas =     id  value
0   1      3
1   2      4
two chunk =     id  value
0   1      3
0   2      4
one chunk =     id  value
0   1    3.0
1   2    NaN
0   1    NaN
1   2    4.0
0   1    NaN
1   2    NaN
0   1    NaN
1   2    NaN
```

**Additional context**

In merge op when the left or right operand has just one chunk, we avoid to do a shuffle merge, but the current implentation unfornately yeilds different result.

The bug was found by reading the code of `merge` op.


Discussion:

Resolving patch:
diff --git a/mars/dataframe/merge/merge.py b/mars/dataframe/merge/merge.py
--- a/mars/dataframe/merge/merge.py
+++ b/mars/dataframe/merge/merge.py
@@ -370,7 +370,9 @@ def tile(cls, op):
         left = build_concatenated_rows_frame(op.inputs[0])
         right = build_concatenated_rows_frame(op.inputs[1])
 
-        if len(left.chunks) == 1 or len(right.chunks) == 1:
+        if (len(left.chunks) == 1 and op.how in ["right", "inner"]) or (
+            len(right.chunks) == 1 and op.how in ["left", "inner"]
+        ):
             return cls._tile_one_chunk(op, left, right)
 
         left_row_chunk_size = left.chunk_shape[0]


Test patch:
diff --git a/mars/dataframe/merge/tests/test_merge_execution.py b/mars/dataframe/merge/tests/test_merge_execution.py
--- a/mars/dataframe/merge/tests/test_merge_execution.py
+++ b/mars/dataframe/merge/tests/test_merge_execution.py
@@ -340,6 +340,20 @@ def test_merge_one_chunk(setup):
         result.sort_values(by=result.columns[1]).reset_index(drop=True),
     )
 
+    # left have one chunk and how="left", then one chunk tile
+    # will results in wrong results, see #GH 2107
+    mdf1 = from_pandas(df1, chunk_size=2)
+    mdf2 = from_pandas(df2)
+
+    expected = df2.merge(df1, left_on="rkey", right_on="lkey", how="left")
+    jdf = mdf2.merge(mdf1, left_on="rkey", right_on="lkey", how="left")
+    result = jdf.execute().fetch()
+
+    pd.testing.assert_frame_equal(
+        expected.sort_values(by=expected.columns[1]).reset_index(drop=True),
+        result.sort_values(by=result.columns[1]).reset_index(drop=True),
+    )
+
 
 def test_merge_on_duplicate_columns(setup):
     raw1 = pd.DataFrame(