internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ros/ros_comm

Issue:
[roslaunch] --ros-args option fails in noetic
# Short description

Running `roslaunch` with the option `--ros-args` under ROS noetic fails.

# Expected behavior

Print the arguments of the launch file like under ROS melodic.
As an example, getting the arguments of roslaunch's own example launch-file via
```
docker run --rm -it ros:melodic-robot roslaunch roslaunch example.launch --ros-args
```
gives
```
Optional Arguments:
  commandline_true_arg (default "true"): undocumented
```

# Acutual behavior

Running any of the below commands
```
docker run --rm -it ros:noetic-robot roslaunch roslaunch example.launch --ros-args
docker run --rm -it ros:noetic-robot roslaunch --ros-args roslaunch example.launch
docker run --rm -it ros:noetic-robot roslaunch /opt/ros/noetic/share/roslaunch/resources/example.launch --ros-args
docker run --rm -it ros:noetic-robot roslaunch --ros-args /opt/ros/noetic/share/roslaunch/resources/example.launch
```

Gives 
```
Traceback (most recent call last):
  File "/opt/ros/noetic/lib/python3/dist-packages/roslaunch/__init__.py", line 275, in main
    import arg_dump as roslaunch_arg_dump
ModuleNotFoundError: No module named 'arg_dump'
```

# Thoughts

I suspect that https://github.com/ros/ros_comm/blob/7ee09ea5cc215733014f47394e6c55ccff69d814/tools/roslaunch/src/roslaunch/__init__.py#L277

should be 
```python
from . import arg_dump as roslaunch_arg_dump 
```
because of the switch to python 3.


Discussion:
Please create a pull request with the proposed change.

Resolving patch:
diff --git a/tools/roslaunch/src/roslaunch/__init__.py b/tools/roslaunch/src/roslaunch/__init__.py
--- a/tools/roslaunch/src/roslaunch/__init__.py
+++ b/tools/roslaunch/src/roslaunch/__init__.py
@@ -274,7 +274,7 @@ def main(argv=sys.argv):
             elif options.file_list:
                 rlutil.print_file_list(args)
             elif options.ros_args:
-                import arg_dump as roslaunch_arg_dump
+                from . import arg_dump as roslaunch_arg_dump
                 roslaunch_arg_dump.dump_args(args)
             else:
                 node_args.print_node_list(args)


Test patch:
Repository: ros/ros_comm

Issue:
Rospy import *
Hi, 

Doing 
```python
from rospy import *
```
raises the following exception : 
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'rospy' has no attribute 'NodeProxy'
```
After some investigations, `NodeProxy` doesn't seem to exist anymore in the codebase. Simply removing it from the exports should do the trick.



Discussion:
Please consider to contribute a pull request for the suggested fix.

Resolving patch:
diff --git a/clients/rospy/src/rospy/__init__.py b/clients/rospy/src/rospy/__init__.py
--- a/clients/rospy/src/rospy/__init__.py
+++ b/clients/rospy/src/rospy/__init__.py
@@ -113,7 +113,6 @@
     'logerr_once', 'logfatal_once',
     'parse_rosrpc_uri',
     'MasterProxy',
-    'NodeProxy',    
     'ROSException',
     'ROSSerializationException',
     'ROSInitException',


Test patch:
Repository: ros/ros_comm

Issue:
NameError on Python 3
If I run `rostopic hz` with two or more topics as arguments, it crashes:
```python
Traceback (most recent call last):
  File "/opt/ros/noetic/bin/rostopic", line 35, in <module>
    rostopic.rostopicmain()
  File "/opt/ros/noetic/lib/python3/dist-packages/rostopic/__init__.py", line 2132, in rostopicmain
    _rostopic_cmd_hz(argv)
  File "/opt/ros/noetic/lib/python3/dist-packages/rostopic/__init__.py", line 1523, in _rostopic_cmd_hz
    _rostopic_hz(topics, window_size=window_size, filter_expr=filter_expr,
  File "/opt/ros/noetic/lib/python3/dist-packages/rostopic/__init__.py", line 323, in _rostopic_hz
    rt.print_hz(topics)
  File "/opt/ros/noetic/lib/python3/dist-packages/rostopic/__init__.py", line 270, in print_hz
    print(_get_ascii_table(header, stats))
  File "/opt/ros/noetic/lib/python3/dist-packages/rostopic/__init__.py", line 285, in _get_ascii_table
    body = '\n'.join('   '.join(cols[h][i] for h in header) for i in xrange(n_rows))
NameError: name 'xrange' is not defined
```

I get this in ROS noetic with Python 3.8 and with Melodic with Python 3.6. xrange is not supported anymore in Python 3. `rostopic hz` with just one topic works fine, because it does not print the ASCII table.


Discussion:

Resolving patch:
diff --git a/tools/rostopic/src/rostopic/__init__.py b/tools/rostopic/src/rostopic/__init__.py
--- a/tools/rostopic/src/rostopic/__init__.py
+++ b/tools/rostopic/src/rostopic/__init__.py
@@ -282,7 +282,7 @@ def _get_ascii_table(header, cols):
     # sum of col and each 3 spaces width
     table_width = sum(col_widths) + 3 * (len(header) - 1)
     n_rows = len(cols[header[0]])
-    body = '\n'.join('   '.join(cols[h][i] for h in header) for i in xrange(n_rows))
+    body = '\n'.join('   '.join(cols[h][i] for h in header) for i in range(n_rows))
     table = '{header}\n{hline}\n{body}\n'.format(
         header='   '.join(header_aligned), hline='=' * table_width, body=body)
     return table


Test patch:
Repository: ros/ros_comm

Issue:
non persistent rospy ServiceProxy is not thread-safe
Relevant code

https://github.com/ros/ros_comm/blob/090daa56f57d18c39946f107bcee5e3efefb2cac/clients/rospy/src/rospy/impl/tcpros_service.py#L481-L537


The self.transport gets set during a service proxy call, and then gets unset if the proxy is not persistent. 

This behavior is not thread safe if multiple threads are trying to use the same service proxy, as it could get unset as another thread is about to use the self.transport variable.


running the following should lead to errors
```python
     #!/usr/bin/env python

    import rospy
    from std_srvs.srv import Empty, EmptyResponse
    from threading import Timer

    class Test():
        def __init__(self):
            rospy.Service('/test', Empty, self.handle_request)
            self.proxy = rospy.ServiceProxy('/test', Empty, persistent=False)
        
        def handle_request(self, request):
            rospy.sleep(0.1)
            return EmptyResponse()
        
        def call_service(self):
            response = self.proxy.call()
            rospy.logwarn(response)
        
    if __name__ == "__main__":
        rospy.init_node("test")
        t = Test()
        a1 = Timer(2.01, t.call_service)
        a2 = Timer(2.02, t.call_service)
        a3 = Timer(2.03, t.call_service)
        a4 = Timer(2.04, t.call_service)
        a5 = Timer(2.05, t.call_service)
        a1.start()
        a2.start()
        a3.start()
        a4.start()
        a5.start()  
        rospy.spin()
```



Discussion:
i use anaconda in windows64 . python 3
line1: import rospy -> dosen work for me. and this error happen: 
(ModuleNotFoundError: No module named 'rospy')
how an i install rospy module in anaconda 

Resolving patch:
diff --git a/clients/rospy/src/rospy/impl/tcpros_service.py b/clients/rospy/src/rospy/impl/tcpros_service.py
--- a/clients/rospy/src/rospy/impl/tcpros_service.py
+++ b/clients/rospy/src/rospy/impl/tcpros_service.py
@@ -510,7 +510,8 @@ def call(self, *args, **kwds):
             except TransportInitError as e:
                 # can be a connection or md5sum mismatch
                 raise ServiceException("unable to connect to service: %s"%e)
-            self.transport = transport
+            if self.persistent:
+                self.transport = transport
         else:
             transport = self.transport
 


Test patch:
Repository: vllm-project/vllm

Issue:
Decrease the default size of swap space
The current default swap space size (20 GiB per GPU) is a bit too large. It can lead to OOM especially for the machine with multiple GPUs.


Discussion:
Will be addressed after #81 

Resolving patch:
diff --git a/cacheflow/config.py b/cacheflow/config.py
--- a/cacheflow/config.py
+++ b/cacheflow/config.py
@@ -3,6 +3,11 @@
 import torch
 from transformers import AutoConfig, PretrainedConfig
 
+from cacheflow.logger import init_logger
+from cacheflow.utils import get_cpu_memory
+
+logger = init_logger(__name__)
+
 _GiB = 1 << 30
 
 
@@ -73,11 +78,37 @@ def __init__(
         self.block_size = block_size
         self.gpu_memory_utilization = gpu_memory_utilization
         self.swap_space_bytes = swap_space * _GiB
+        self._verify_args()
 
         # Will be set after profiling.
         self.num_gpu_blocks = None
         self.num_cpu_blocks = None
 
+    def _verify_args(self) -> None:
+        if self.gpu_memory_utilization > 1.0:
+            raise ValueError(
+                "GPU memory utilization must be less than 1.0. Got "
+                f"{self.gpu_memory_utilization}.")
+
+    def verify_with_parallel_config(
+        self,
+        parallel_config: "ParallelConfig",
+    ) -> None:
+        total_cpu_memory = get_cpu_memory()
+        # FIXME(woosuk): Here, it is assumed that the GPUs in a tensor parallel
+        # group are in the same node. However, the GPUs may span multiple nodes.
+        num_gpus_per_node = parallel_config.tensor_parallel_size
+        cpu_memory_usage = self.swap_space_bytes * num_gpus_per_node
+
+        msg = (
+            f"{cpu_memory_usage / _GiB:.2f} GiB out of "
+            f"the {total_cpu_memory / _GiB:.2f} GiB total CPU memory is "
+            "allocated for the swap space.")
+        if cpu_memory_usage > 0.7 * total_cpu_memory:
+            raise ValueError("Too large swap space. " + msg)
+        elif cpu_memory_usage > 0.4 * total_cpu_memory:
+            logger.warn("Possibly too large swap space. " + msg)
+
 
 class ParallelConfig:
 
diff --git a/cacheflow/server/llm_server.py b/cacheflow/server/llm_server.py
--- a/cacheflow/server/llm_server.py
+++ b/cacheflow/server/llm_server.py
@@ -84,6 +84,7 @@ def __init__(
 
     def _verify_args(self) -> None:
         self.model_config.verify_with_parallel_config(self.parallel_config)
+        self.cache_config.verify_with_parallel_config(self.parallel_config)
 
     def _init_cache(self) -> None:
         # Get the maximum number of blocks that can be allocated on GPU and CPU.
diff --git a/cacheflow/utils.py b/cacheflow/utils.py
--- a/cacheflow/utils.py
+++ b/cacheflow/utils.py
@@ -24,8 +24,10 @@ def reset(self) -> None:
 
 
 def get_gpu_memory(gpu: int = 0) -> int:
+    """Returns the total memory of the GPU in bytes."""
     return torch.cuda.get_device_properties(gpu).total_memory
 
 
 def get_cpu_memory() -> int:
+    """Returns the total CPU memory of the node in bytes."""
     return psutil.virtual_memory().total


Test patch:
Repository: vllm-project/vllm

Issue:
Write README



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -165,7 +165,7 @@ def get_requirements() -> List[str]:
         "Topic :: Scientific/Engineering :: Artificial Intelligence",
     ],
     packages=setuptools.find_packages(
-        exclude=("benchmarks", "csrc", "docs", "examples", "tests")),
+        exclude=("assets", "benchmarks", "csrc", "docs", "examples", "tests")),
     python_requires=">=3.8",
     install_requires=get_requirements(),
     ext_modules=ext_modules,


Test patch:
Repository: vllm-project/vllm

Issue:
Documentation on running basic python server and FastAPI server



Discussion:

Resolving patch:
diff --git a/vllm/outputs.py b/vllm/outputs.py
--- a/vllm/outputs.py
+++ b/vllm/outputs.py
@@ -4,6 +4,18 @@
 
 
 class CompletionOutput:
+    """The output data of one completion output of a request.
+
+    Args:
+        index: The index of the output in the request.
+        text: The generated output text.
+        token_ids: The token IDs of the generated output text.
+        cumulative_logprob: The cumulative log probability of the generated
+            output text.
+        logprobs: The log probabilities of the top probability words at each
+            position if the logprobs are requested.
+        finish_reason: The reason why the sequence is finished.
+    """
 
     def __init__(
         self,
@@ -11,7 +23,7 @@ def __init__(
         text: str,
         token_ids: List[int],
         cumulative_logprob: float,
-        logprobs: List[Dict[int, float]],
+        logprobs: Optional[List[Dict[int, float]]],
         finish_reason: Optional[str] = None,
     ) -> None:
         self.index = index
@@ -34,7 +46,14 @@ def __repr__(self) -> str:
 
 
 class RequestOutput:
+    """The output data of a request to the LLM.
 
+    Args:
+        request_id: The unique ID of the request.
+        prompt: The prompt string of the request.
+        prompt_token_ids: The token IDs of the prompt.
+        outputs: The output sequences of the request.
+    """
     def __init__(
         self,
         request_id: str,


Test patch:
Repository: vllm-project/vllm

Issue:
Provide proper error message when OOM happened during memory profiling
To prevent the confusion in #184 


Discussion:

Resolving patch:
diff --git a/vllm/engine/llm_engine.py b/vllm/engine/llm_engine.py
--- a/vllm/engine/llm_engine.py
+++ b/vllm/engine/llm_engine.py
@@ -127,6 +127,12 @@ def _init_cache(self) -> None:
         # FIXME(woosuk): Change to debug log.
         logger.info(f'# GPU blocks: {num_gpu_blocks}, '
                     f'# CPU blocks: {num_cpu_blocks}')
+
+        if num_gpu_blocks <= 0 or num_cpu_blocks <= 0:
+            raise ValueError("No available memory for the cache blocks. "
+                             "Try increasing `gpu_memory_utilization` when "
+                             "initializing the engine.")
+
         self.cache_config.num_gpu_blocks = num_gpu_blocks
         self.cache_config.num_cpu_blocks = num_cpu_blocks
 
diff --git a/vllm/outputs.py b/vllm/outputs.py
--- a/vllm/outputs.py
+++ b/vllm/outputs.py
@@ -53,6 +53,7 @@ class RequestOutput:
         prompt: The prompt string of the request.
         prompt_token_ids: The token IDs of the prompt.
         outputs: The output sequences of the request.
+        finished: Whether the whole request is finished.
     """
     def __init__(
         self,


Test patch:
Repository: vllm-project/vllm

Issue:
On the way scheduler calculate the number of sequences
In version 0.1.1, the scheduler calculates the number of sequences it schedules like the following:
```python
num_new_seqs = seq_group.num_seqs(status=SequenceStatus.SWAPPED)
num_curr_seqs = len(self.running)
if num_curr_seqs + num_new_seqs > self.scheduler_config.max_num_seqs:
    break
```
`num_curr_seqs` is the number of sequence groups while `num_new_seqs` stands for the number of sequences in the new sequence group, I think it cannot form a meaningful metric by adding them together.


Discussion:
@LinPoly You're right. This indeed seems like a bug. Thanks for reporting this and please keep exploring vLLM!

Resolving patch:
diff --git a/vllm/core/scheduler.py b/vllm/core/scheduler.py
--- a/vllm/core/scheduler.py
+++ b/vllm/core/scheduler.py
@@ -156,7 +156,9 @@ def _schedule(self) -> Tuple[SchedulerOutputs, List[str]]:
             # The total number of sequences in the RUNNING state should not
             # exceed the maximum number of sequences.
             num_new_seqs = seq_group.num_seqs(status=SequenceStatus.SWAPPED)
-            num_curr_seqs = len(self.running)
+            num_curr_seqs = sum(
+                seq_group.num_seqs(status=SequenceStatus.RUNNING)
+                for seq_group in self.running)
             if num_curr_seqs + num_new_seqs > self.scheduler_config.max_num_seqs:
                 break
 
@@ -198,7 +200,9 @@ def _schedule(self) -> Tuple[SchedulerOutputs, List[str]]:
                 # The total number of sequences in the RUNNING state should not
                 # exceed the maximum number of sequences.
                 num_new_seqs = seq_group.num_seqs(status=SequenceStatus.WAITING)
-                num_curr_seqs = len(self.running)
+                num_curr_seqs = sum(
+                    seq_group.num_seqs(status=SequenceStatus.RUNNING)
+                    for seq_group in self.running)
                 if num_curr_seqs + num_new_seqs > self.scheduler_config.max_num_seqs:
                     break
 


Test patch:
Repository: vllm-project/vllm

Issue:
Input and output orders are different
I was trying to do batch inference by feeding a list of texts into `llm.generate()`. But I found that the outputs have different orders from examples in the input list. For example, the result for the first example in the input list may be the second in the output list. Anything suggestions that I may have done something wrong? Thanks!


Discussion:

Resolving patch:
diff --git a/vllm/entrypoints/llm.py b/vllm/entrypoints/llm.py
--- a/vllm/entrypoints/llm.py
+++ b/vllm/entrypoints/llm.py
@@ -155,4 +155,8 @@ def _run_engine(self, use_tqdm: bool) -> List[RequestOutput]:
                         pbar.update(1)
         if use_tqdm:
             pbar.close()
+        # Sort the outputs by request ID.
+        # This is necessary because some requests may be finished earlier than
+        # its previous requests.
+        outputs = sorted(outputs, key=lambda x: int(x.request_id))
         return outputs


Test patch:
Repository: vllm-project/vllm

Issue:
RayWorker import error
When import `vllm` I got following error
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/code/vllm/vllm/__init__.py", line 4, in <module>
    from vllm.engine.async_llm_engine import AsyncLLMEngine
  File "/code/vllm/vllm/engine/async_llm_engine.py", line 7, in <module>
    from vllm.engine.llm_engine import LLMEngine
  File "/code/vllm/vllm/engine/llm_engine.py", line 9, in <module>
    from vllm.engine.ray_utils import initialize_cluster, ray, RayWorker
ImportError: cannot import name 'RayWorker' from 'vllm.engine.ray_utils' 
```
It seems `ray` requires `pandas` I haven't installed it.



Discussion:

Resolving patch:
diff --git a/vllm/engine/ray_utils.py b/vllm/engine/ray_utils.py
--- a/vllm/engine/ray_utils.py
+++ b/vllm/engine/ray_utils.py
@@ -27,6 +27,7 @@ def execute_method(self, method, *args, **kwargs):
 except ImportError:
     ray = None
     TorchDistributedWorker = None
+    RayWorker = None  # pylint: disable=invalid-name
 
 if TYPE_CHECKING:
     from ray.util.placement_group import PlacementGroup


Test patch:
Repository: vllm-project/vllm

Issue:
Issue while loading MPT-7B-8K-instruct
Newbie Question:

I thought support for all MPT models is enabled by default, but when I try to load 8K variant of the MPT-7B model. I'm running into the following error while instantiation:
```
ValueError: Invalid shape for attention bias: torch.Size([32, 10, 10]) (expected (1, 32, 10, 10))
  query.shape: torch.Size([1, 10, 32, 128])
  key.shape  : torch.Size([1, 10, 32, 128])
  value.shape: torch.Size([1, 10, 32, 128])
```

Can someone help/ guide me on this issue?


Discussion:
Hi @ramkrithik, thanks for reporting the bug. It seems the bug is due to the new release of `xformers`, which includes some breaking changes. I'm fixing this issue.

Resolving patch:
diff --git a/vllm/model_executor/layers/attention.py b/vllm/model_executor/layers/attention.py
--- a/vllm/model_executor/layers/attention.py
+++ b/vllm/model_executor/layers/attention.py
@@ -357,11 +357,12 @@ def set_attn_bias(self, input_metadata: InputMetadata) -> None:
             # be sliced from a tensor whose length is a multiple of 8.
             padded_len = (prompt_len + 7) // 8 * 8
             bias = torch.empty(
+                1,  # batch_size
                 self.num_heads,
-                padded_len,
+                prompt_len,
                 padded_len,
                 device=self.alibi_slopes.device,
-            )[:, :prompt_len, :prompt_len].copy_(bias)
+            )[:, :, :, :prompt_len].copy_(bias)
             bias.mul_(self.alibi_slopes[:, None, None])
             attn_bias = LowerTriangularMaskWithTensorBias(bias)
             input_metadata.attn_bias.append(attn_bias)


Test patch:
Repository: vllm-project/vllm

Issue:
AWQ does not support Turing GPUs
              @WoosukKwon Doesn't it support Turing arch? my GPU's compute capabitlity is 7.5. CUDA-12.1.

build Error message:

  `ptxas /tmp/tmpxft_0006e7c4_00000000-6_gemm_kernels.ptx, line 928; error   : Feature '.m16n8k16' requires .target sm_80 or higher`

If not, hope can add backward compatibility for kernel build.

_Originally posted by @esmeetu in https://github.com/vllm-project/vllm/issues/1032#issuecomment-1722179620_
            


Discussion:

Resolving patch:
diff --git a/vllm/model_executor/model_loader.py b/vllm/model_executor/model_loader.py
--- a/vllm/model_executor/model_loader.py
+++ b/vllm/model_executor/model_loader.py
@@ -68,6 +68,14 @@ def get_model(model_config: ModelConfig) -> nn.Module:
         quant_config = get_quant_config(model_config.quantization,
                                         model_config.model,
                                         model_config.download_dir)
+        capability = torch.cuda.get_device_capability()
+        capability = capability[0] * 10 + capability[1]
+        if capability < quant_config.get_min_capability():
+            raise ValueError(
+                f"The quantization method {model_config.quantization} is not "
+                "supported for the current GPU. "
+                f"Minimum capability: {quant_config.get_min_capability()}. "
+                f"Current capability: {capability}.")
         supported_dtypes = quant_config.get_supported_act_dtypes()
         if model_config.dtype not in supported_dtypes:
             raise ValueError(
diff --git a/vllm/model_executor/quantization_utils/awq.py b/vllm/model_executor/quantization_utils/awq.py
--- a/vllm/model_executor/quantization_utils/awq.py
+++ b/vllm/model_executor/quantization_utils/awq.py
@@ -40,6 +40,11 @@ def get_name(cls) -> str:
     def get_supported_act_dtypes(cls) -> List[torch.dtype]:
         return [torch.half]
 
+    @classmethod
+    def get_min_capability(cls) -> int:
+        # The AWQ kernel only supports Ampere or newer GPUs.
+        return 80
+
     @classmethod
     def get_config_filenames(cls) -> List[str]:
         return [
diff --git a/vllm/model_executor/quantization_utils/base.py b/vllm/model_executor/quantization_utils/base.py
--- a/vllm/model_executor/quantization_utils/base.py
+++ b/vllm/model_executor/quantization_utils/base.py
@@ -15,6 +15,16 @@ def get_supported_act_dtypes(cls) -> List[torch.dtype]:
         """List of supported activation dtypes."""
         raise NotImplementedError
 
+    @classmethod
+    def get_min_capability(cls) -> int:
+        """Minimum GPU capability to support the quantization method.
+
+        E.g., 70 for Volta, 75 for Turing, 80 for Ampere.
+        This requirement is due to the custom CUDA kernels used by the
+        quantization method.
+        """
+        raise NotImplementedError
+
     @classmethod
     def get_config_filenames(cls) -> List[str]:
         """List of filenames to search for in the model directory."""


Test patch:
Repository: vllm-project/vllm

Issue:
[v0.2.0] Release Tracker
## Major changes

* Up to 60% performance improvement by optimizing de-tokenization and sampler
* Initial support for AWQ (performance not optimized)
* Support for RoPE scaling and LongChat
* Support for Mistral-7B

## PRs to be merged before the release

- [x] Vectorized sampler: #1048, #820 
- [x] LongChat: #555 
- [x] `TORCH_CUDA_ARCH_LIST` build option: #1074 
- [x] Support for Mistral-7B: #1196 
- [x] #1198  
- ~~[ ] FP32 RoPE kernel: #1061~~ (deferred to the next PR)


Discussion:

Resolving patch:
diff --git a/vllm/__init__.py b/vllm/__init__.py
--- a/vllm/__init__.py
+++ b/vllm/__init__.py
@@ -8,7 +8,7 @@
 from vllm.outputs import CompletionOutput, RequestOutput
 from vllm.sampling_params import SamplingParams
 
-__version__ = "0.1.7"
+__version__ = "0.2.0"
 
 __all__ = [
     "LLM",


Test patch: