internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pyjanitor-devs/pyjanitor

Issue:
Bug with complement in `filter_on` with multiple criteria
# Brief Description

`filter_on` function does not return the complement properly. (see MWE below) Came across this bug when I was writing the MWE docstring for it.

# Minimally Reproducible Code

```python
import janitor
import pandas as pd
df = pd.DataFrame({
    "student_id": ["S1", "S2", "S3"],
    "score": [40, 60, 85],
})

df.filter_on("(score < 50) & (score > 70)", complement=True)
# the above should return students whose scores are between 50 and 70 included.
# but the result is

  student_id  score
2         S3     85
```

Reason is simple, see the source code implementation:

```python
if complement:
    return df.query("not " + criteria)
```

criteria should be enclosed with parenthesis to ensure complement of the whole criteria is taken instead.


Perhaps a test should also be added to cover this case as well (as in, multiple criteria).


Discussion:

Resolving patch:
diff --git a/janitor/functions/filter.py b/janitor/functions/filter.py
--- a/janitor/functions/filter.py
+++ b/janitor/functions/filter.py
@@ -107,7 +107,7 @@ def filter_on(
     :returns: A filtered pandas DataFrame.
     """
     if complement:
-        return df.query("not " + criteria)
+        return df.query(f"not ({criteria})")
     return df.query(criteria)
 
 
diff --git a/janitor/io.py b/janitor/io.py
--- a/janitor/io.py
+++ b/janitor/io.py
@@ -89,15 +89,16 @@ def read_commandline(cmd: str, **kwargs) -> pd.DataFrame:
     ```python
     import janitor as jn
     df = jn.read_commandline("cat data.csv | grep .SEA1AA")
+    ```
 
     This function assumes that your command line command will return
-    an output that is parsable using pandas.read_csv and StringIO.
-    We default to using pd.read_csv underneath the hood.
+    an output that is parsable using `pandas.read_csv` and StringIO.
+    We default to using `pd.read_csv` underneath the hood.
     Keyword arguments are passed through to read_csv.
-    ```
 
     :param cmd: Shell command to preprocess a file on disk.
-    :param kwargs: Keyword arguments that are passed through to pd.read_csv().
+    :param kwargs: Keyword arguments that are passed through to
+        `pd.read_csv()`.
     :raises JanitorError: If commandline command is malformed or invalid.
     :returns: A pandas DataFrame parsed from the stdout of the underlying
         shell.


Test patch:
diff --git a/tests/functions/test_filter_on.py b/tests/functions/test_filter_on.py
--- a/tests/functions/test_filter_on.py
+++ b/tests/functions/test_filter_on.py
@@ -6,3 +6,10 @@
 def test_filter_on(dataframe, complement, expected):
     df = dataframe.filter_on("a == 3", complement=complement)
     assert len(df) == expected
+
+
+@pytest.mark.functions
+@pytest.mark.parametrize("complement,expected", [(True, 3), (False, 6)])
+def test_filter_on_with_multiple_criteria(dataframe, complement, expected):
+    df = dataframe.filter_on("(a == 3) | (a == 1)", complement=complement)
+    assert len(df) == expected
Repository: pyjanitor-devs/pyjanitor

Issue:
duplicated columns generate more rows than required in `pivot_longer`
# Brief Description

If the columns are duplicated, more rows are generated than necessary (duplicated) when melted.

# System Information

<!-- System information helps us. To keep things simple, just let us know the OS and Python version first.
You can provide the optional information later. -->

- Operating system: macOS
- Python version (required): 3.8

# Minimally Reproducible Code

copied from [SO](https://stackoverflow.com/q/72126060/7175713)
```
rows = [['credit', 1, 1, 2, 3]]
columns = ['Type', 'amount', 'active', 'amount', 'active']

df = pd.DataFrame(rows, columns=columns)
df = df.set_index('Type')


        amount  active  amount  active
Type                                   
credit       1       1       2       3


# output from pivot_longer
df.pivot_longer(names_to = ['amount', 'active'], names_pattern = ['amount', 'active'])
 
   amount  active
0       1       1
1       2       3
2       1       1
3       2       3

# expected output

        amount  active
Type
credit       1       1
credit       2       3
```





Discussion:

Resolving patch:
diff --git a/janitor/functions/pivot.py b/janitor/functions/pivot.py
--- a/janitor/functions/pivot.py
+++ b/janitor/functions/pivot.py
@@ -606,6 +606,8 @@ def _computations_pivot_longer(
             name: extract_array(df[name], extract_numpy=True) for name in index
         }
 
+    if len(column_names) != len(set(column_names)):
+        column_names = pd.unique(column_names)
     df = df.loc[:, column_names]
 
     len_index = len(df)


Test patch:
diff --git a/tests/functions/test_pivot_longer.py b/tests/functions/test_pivot_longer.py
--- a/tests/functions/test_pivot_longer.py
+++ b/tests/functions/test_pivot_longer.py
@@ -1156,3 +1156,22 @@ def test_names_transform_numeric():
     ).reset_index()
 
     assert_frame_equal(actual, expected)
+
+
+def test_duplicated_columns():
+    """Test output for duplicated columns."""
+    rows = [["credit", 1, 1, 2, 3]]
+    columns = ["Type", "amount", "active", "amount", "active"]
+
+    df = pd.DataFrame(rows, columns=columns)
+    df = df.set_index("Type")
+
+    actual = pd.DataFrame(
+        {"amount": [1, 2], "active": [1, 3]},
+        index=pd.Index(["credit", "credit"], name="Type"),
+    )
+    expected = df.pivot_longer(
+        names_to=".value", names_pattern="(.+)", ignore_index=False
+    )
+
+    assert_frame_equal(actual, expected)
Repository: pyjanitor-devs/pyjanitor

Issue:
string selection on multiindex top level
# Brief Description

string selection does not work on MultiIndex top level

# System Information

<!-- System information helps us. To keep things simple, just let us know the OS and Python version first.
You can provide the optional information later. -->

- Operating system: linux
- OS details (optional): popos
- Python version (required):3.10

# Minimally Reproducible Code

```
df = pd.DataFrame(
    np.zeros((2,4)),
    columns=pd.MultiIndex.from_product([('a','b'),(1,2)])
)

     a         b     
     1    2    1    2
0  0.0  0.0  0.0  0.0
1  0.0  0.0  0.0  0.0

df.select_columns('a')

    358             return _select_regex(
    359                 index, (arg, fnmatch.translate(arg)), source="fnmatch"
    360             )
    361         raise KeyError(f"No match was returned for '{arg}'") from exc
--> 362 raise KeyError(f"No match was returned for '{arg}'")

KeyError: "No match was returned for 'a'"
```


Discussion:
something seems to have changed in pandas 2.0 with the `is_string_dtype` check

Resolving patch:
diff --git a/janitor/functions/utils.py b/janitor/functions/utils.py
--- a/janitor/functions/utils.py
+++ b/janitor/functions/utils.py
@@ -338,6 +338,8 @@ def _index_dispatch(arg, df, axis):  # noqa: F811
     or a slice.
     """
     index = getattr(df, axis)
+    if isinstance(index, pd.MultiIndex):
+        index = index.get_level_values(0)
     if _is_str_or_cat(index) or is_datetime64_dtype(index):
         try:
             return index.get_loc(arg)
@@ -345,8 +347,6 @@ def _index_dispatch(arg, df, axis):  # noqa: F811
             if _is_str_or_cat(index):
                 if arg == "*":
                     return slice(None)
-                if isinstance(index, pd.MultiIndex):
-                    index = index.get_level_values(0)
                 # label selection should be case sensitive
                 # fix for Github Issue 1160
                 # translating to regex solves the case sensitivity


Test patch:
diff --git a/tests/functions/test_select_columns.py b/tests/functions/test_select_columns.py
--- a/tests/functions/test_select_columns.py
+++ b/tests/functions/test_select_columns.py
@@ -454,3 +454,10 @@ def test_select_groupby(dataframe):
     expected = dataframe.select_dtypes("number").groupby(dataframe["a"]).sum()
     actual = dataframe.groupby("a").pipe(get_columns, is_numeric_dtype).sum()
     assert_frame_equal(expected, actual)
+
+
+def test_select_str_multiindex(multiindex):
+    """Test str selection on a MultiIndex - exact match"""
+    expected = multiindex.select_columns("bar")
+    actual = multiindex.loc(axis=1)[["bar"]]
+    assert_frame_equal(expected, actual)
Repository: xorbitsai/inference

Issue:
BUG: too many clients
### Describe the bug
When running the model_ref.generate() function in iPython, there seems to be a client created for every word generation, eventually leading to the following error:

`gaierror: [Errno 8] nodename nor servname provided, or not known`

### To Reproduce

`python -m plexar.deploy.cmdline supervisor -a localhost:9999 --log-level debug`

`python -m plexar.deploy.cmdline worker --supervisor-address localhost:9999 -a localhost:10000 --log-level debug`

```
import sys
from plexar.client import Client
client = Client("localhost:9999")
model_uid = client.launch_model("wizardlm-v1.0",7,"ggmlv3","q4_0")
model_ref = client.get_model(model_uid)


async for c in await model_ref.generate("Once upon a time, there was a very old computer.", {'max_tokens': 512}): sys.stdout.write(c['choices'][0]['text'])
```

### Expected behavior
First the warnings are printed: `Actor caller has created too many clients ([some number] >= 100), the global router may not be set.`

Then we have the gaierror after the `[some number]` exceeds 240.


Discussion:

Resolving patch:
diff --git a/plexar/__init__.py b/plexar/__init__.py
--- a/plexar/__init__.py
+++ b/plexar/__init__.py
@@ -19,8 +19,12 @@
 
 
 def install():
+    from xoscar.backends.router import Router
+
     from .model import install as install_model
 
+    default_router = Router.get_instance_or_empty()
+    Router.set_instance(default_router)
     install_model()
 
 


Test patch:
Repository: xorbitsai/inference

Issue:
BUG: worker timeout when downloading a model



Discussion:

Resolving patch:
diff --git a/xinference/core/service.py b/xinference/core/service.py
--- a/xinference/core/service.py
+++ b/xinference/core/service.py
@@ -90,7 +90,6 @@ async def _choose_worker(self) -> xo.ActorRefType["WorkerActor"]:
 
         raise RuntimeError("No available worker found")
 
-    @log
     async def launch_builtin_model(
         self,
         model_uid: str,
@@ -100,10 +99,22 @@ async def launch_builtin_model(
         quantization: Optional[str],
         **kwargs,
     ) -> xo.ActorRefType["ModelActor"]:
+        logger.debug(
+            (
+                f"Enter launch_builtin_model, model_uid: %s, model_name: %s, model_size: %s, "
+                f"model_format: %s, quantization: %s"
+            ),
+            model_uid,
+            model_name,
+            str(model_size_in_billions) if model_size_in_billions else "",
+            model_format,
+            quantization,
+        )
+
         assert model_uid not in self._model_uid_to_worker
 
         worker_ref = await self._choose_worker()
-        model_ref = await worker_ref.launch_builtin_model(
+        model_ref = yield worker_ref.launch_builtin_model(
             model_uid=model_uid,
             model_name=model_name,
             model_size_in_billions=model_size_in_billions,
@@ -111,9 +122,10 @@ async def launch_builtin_model(
             quantization=quantization,
             **kwargs,
         )
+        # TODO: not protected.
         self._model_uid_to_worker[model_uid] = worker_ref
 
-        return model_ref
+        raise xo.Return(model_ref)
 
     async def _check_dead_nodes(self):
         while True:
@@ -250,10 +262,13 @@ async def launch_builtin_model(
             if model_spec is None:
                 continue
 
-            cls = model_family.cls
-            save_path = model_family.cache(
-                model_spec.model_size_in_billions, model_spec.quantization
+            save_path = await asyncio.to_thread(
+                model_family.cache,
+                model_spec.model_size_in_billions,
+                model_spec.quantization,
             )
+
+            cls = model_family.cls
             model = cls(model_uid, model_spec, save_path, kwargs)
             subpool_address = self._choose_subpool()
             model_ref = await xo.create_actor(


Test patch:
Repository: xorbitsai/inference

Issue:
FEAT: support vicuna-v1.3 33B



Discussion:

Resolving patch:
diff --git a/xinference/model/llm/__init__.py b/xinference/model/llm/__init__.py
--- a/xinference/model/llm/__init__.py
+++ b/xinference/model/llm/__init__.py
@@ -88,12 +88,14 @@ def install():
         else (
             "https://huggingface.co/TheBloke/vicuna-13b-v1.3.0-GGML/resolve/main/"
             f"vicuna-13b-v1.3.0.ggmlv3.{quantization}.bin"
+            if model_size == 13
+            else f"https://huggingface.co/TheBloke/vicuna-33B-GGML/resolve/main/vicuna-33b.ggmlv3.{quantization}.bin"
         )
     )
     MODEL_FAMILIES.append(
         ModelFamily(
             model_name="vicuna-v1.3",
-            model_sizes_in_billions=[7, 13],
+            model_sizes_in_billions=[7, 13, 33],
             model_format="ggmlv3",
             quantizations=[
                 "q2_K",


Test patch:
Repository: xorbitsai/inference

Issue:
BUG: UnicodeDecodeError caused by xinference/model/llm/llm_family.json
### Describe the bug
Windows cmd runned xinference found Error: "**UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 8905: illegal multibyte sequence**"

### To Reproduce
To help us to reproduce this bug, please provide information below:

1. python version: 3.10
2. xinference version: v0.1.0

### Expected behavior
Edit the file: xinference/model/llm/__init__.py
Line number: 129
Add arg encoding: open(json_path, encoding='utf-8')





Discussion:
Thanks for reporting this issue! This problem will be fixed in the next release.

Stay tuned!

Resolving patch:
diff --git a/xinference/model/llm/__init__.py b/xinference/model/llm/__init__.py
--- a/xinference/model/llm/__init__.py
+++ b/xinference/model/llm/__init__.py
@@ -12,6 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import codecs
 import json
 import logging
 import os
@@ -132,5 +133,5 @@ def _install():
     json_path = os.path.join(
         os.path.dirname(os.path.abspath(__file__)), "llm_family.json"
     )
-    for json_obj in json.load(open(json_path)):
+    for json_obj in json.load(codecs.open(json_path, "r", encoding="utf-8")):
         LLM_FAMILIES.append(LLMFamilyV1.parse_obj(json_obj))


Test patch:
Repository: xorbitsai/inference

Issue:
FEAT: Disable Gradio Telemetry
Pull requests are disabled but see here:

https://github.com/arch-btw/inference/pull/1


Discussion:

Resolving patch:
diff --git a/examples/gradio_chatinterface.py b/examples/gradio_chatinterface.py
--- a/examples/gradio_chatinterface.py
+++ b/examples/gradio_chatinterface.py
@@ -105,6 +105,7 @@ def generate_wrapper(message: str, history: List[List[str]]) -> str:
 
     demo = gr.ChatInterface(
         fn=generate_wrapper,
+        analytics_enabled=False,
         examples=[
             "Show me a two sentence horror story with a plot twist",
             "Generate a Haiku poem using trignometry as the central theme",


Test patch:
Repository: xorbitsai/inference

Issue:
BUG: chatglm max_length doesn't work
### Describe the bug
The gen config has `max_tokens`, but chatglm requires `max_length`.

### To Reproduce
To help us to reproduce this bug, please provide information below:

1. Your Python version.
2. The version of xinference you use.
3. Versions of crucial packages.
4. Full stack of the error.
5. Minimized code to reproduce the error.

### Expected behavior
A clear and concise description of what you expected to happen.

### Additional context
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/xinference/model/llm/ggml/chatglm.py b/xinference/model/llm/ggml/chatglm.py
--- a/xinference/model/llm/ggml/chatglm.py
+++ b/xinference/model/llm/ggml/chatglm.py
@@ -183,18 +183,23 @@ def chat(
         logger.debug("Full conversation history:\n%s", str(chat_history_list))
 
         generate_config = self._sanitize_generate_config(generate_config)
+        params = dict(generate_config)
+        if "max_tokens" in params:
+            params["max_length"] = params.pop("max_tokens")
 
         assert self._llm is not None
 
-        if generate_config.get("stream", False):
+        if params.pop("stream", False):
             it = self._llm.stream_chat(
                 chat_history_list,
+                **params,
             )
             assert not isinstance(it, str)
             return self._convert_raw_text_chunks_to_chat(it, self.model_uid)
         else:
             c = self._llm.chat(
                 chat_history_list,
+                **params,
             )
             assert not isinstance(c, Iterator)
             return self._convert_raw_text_completion_to_chat(c, self.model_uid)


Test patch:
Repository: xorbitsai/inference

Issue:
FEAT: support  WizardMath v1.0
### Is your feature request related to a problem? Please describe
https://huggingface.co/WizardLM/WizardMath-13B-V1.0




Discussion:

Resolving patch:
diff --git a/xinference/model/llm/utils.py b/xinference/model/llm/utils.py
--- a/xinference/model/llm/utils.py
+++ b/xinference/model/llm/utils.py
@@ -164,6 +164,16 @@ def get_prompt(
             )
             ret += chat_history[-1]["role"] + ":"
             return ret
+        elif prompt_style.style_name == "ADD_COLON_SINGLE_COT":
+            ret = prompt_style.system_prompt + prompt_style.intra_message_sep
+            for message in chat_history:
+                role = message["role"]
+                content = message["content"]
+                if content:
+                    ret += role + ": " + content + prompt_style.intra_message_sep
+                else:
+                    ret += role + ": Let's think step by step."
+            return ret
         else:
             raise ValueError(f"Invalid prompt style: {prompt_style.style_name}")
 


Test patch:
diff --git a/xinference/model/llm/tests/test_utils.py b/xinference/model/llm/tests/test_utils.py
--- a/xinference/model/llm/tests/test_utils.py
+++ b/xinference/model/llm/tests/test_utils.py
@@ -298,6 +298,36 @@ def test_prompt_style_internlm():
     assert expected == actual
 
 
+def test_prompt_style_add_colon_single_cot():
+    prompt_style = PromptStyleV1(
+        style_name="ADD_COLON_SINGLE_COT",
+        system_prompt=(
+            "Below is an instruction that describes a task. Write a response that appropriately "
+            "completes the request."
+        ),
+        roles=["Instruction", "Response"],
+        intra_message_sep="\n\n### ",
+    )
+
+    chat_history = [
+        ChatCompletionMessage(role=prompt_style.roles[0], content="Hi there."),
+        ChatCompletionMessage(
+            role=prompt_style.roles[1], content="Hello, how may I help you?"
+        ),
+    ]
+    expected = (
+        "Below is an instruction that describes a task. Write a response that appropriately "
+        "completes the request."
+        "\n\n### Instruction: Hi there."
+        "\n\n### Response: Hello, how may I help you?"
+        "\n\n### Instruction: Write a poem."
+        "\n\n### Response: Let's think step by step."
+    )
+    assert expected == ChatModelMixin.get_prompt(
+        "Write a poem.", chat_history, prompt_style
+    )
+
+
 def test_is_valid_model_name():
     from ..utils import is_valid_model_name
Repository: xorbitsai/inference

Issue:
QUESTION: Upper bound for the max_token argument
I noticed that there is an upper bound of 2048 tokens placed on the max_token argument in [xinference/core/restful_api.py](https://github.com/xorbitsai/inference/blob/67a06b40ef4ec36448e504fd23526043719211bc/xinference/core/restful_api.py#L39-L41)

https://github.com/xorbitsai/inference/blob/67a06b40ef4ec36448e504fd23526043719211bc/xinference/core/restful_api.py#L39-L41

Why is this necessary? I feel like it would be reasonable to generate a 4000 token long essay with a model with matching context length, especially when we now have models like [InternLM](https://github.com/xorbitsai/inference/blob/67a06b40ef4ec36448e504fd23526043719211bc/xinference/model/llm/llm_family.json#L965-L1007) with 8k context length and [ChatGLM2-32k](https://github.com/xorbitsai/inference/blob/67a06b40ef4ec36448e504fd23526043719211bc/xinference/model/llm/llm_family.json#L450-L483) with 32k context length.


Discussion:

Resolving patch:
diff --git a/xinference/core/restful_api.py b/xinference/core/restful_api.py
--- a/xinference/core/restful_api.py
+++ b/xinference/core/restful_api.py
@@ -37,7 +37,7 @@
 logger = logging.getLogger(__name__)
 
 max_tokens_field = Field(
-    default=128, ge=1, le=2048, description="The maximum number of tokens to generate."
+    default=128, ge=1, le=32768, description="The maximum number of tokens to generate."
 )
 
 temperature_field = Field(


Test patch:
Repository: xorbitsai/inference

Issue:
BUG: Asking to pad but the tokenizer does not have a padding token
### Describe the bug
When using Llama-2 PyTorch model for embedding creation, there's an issue:

----> 1 model.create_embedding("Once upon a time, there was a very old computer")

File ~/inference/xinference/client.py:324, in RESTfulGenerateModelHandle.create_embedding(self, input)
    322 response = requests.post(url, json=request_body)
    323 if response.status_code != 200:
--> 324     raise RuntimeError(
    325         f"Failed to create the embeddings, detail: {response.json()['detail']}"
    326     )
    328 response_data = response.json()
    329 return response_data

RuntimeError: Failed to create the embeddings, detail: [address=127.0.0.1:35477, pid=951303] Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.



Discussion:

Resolving patch:
diff --git a/xinference/model/llm/pytorch/llama_2.py b/xinference/model/llm/pytorch/llama_2.py
--- a/xinference/model/llm/pytorch/llama_2.py
+++ b/xinference/model/llm/pytorch/llama_2.py
@@ -39,6 +39,9 @@ def __init__(
 
     def _load_model(self, kwargs: dict):
         model, tokenizer = super()._load_model(kwargs)
+        # Llama has no pad token by default
+        # https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/docs/source/en/llm_tutorial.md?plain=1#L125
+        tokenizer.pad_token = tokenizer.eos_token
         model.config.eos_token_id = tokenizer.eos_token_id
         model.config.pad_token_id = tokenizer.pad_token_id
         return model, tokenizer
@@ -76,6 +79,9 @@ def __init__(
 
     def _load_model(self, kwargs: dict):
         model, tokenizer = super()._load_model(kwargs)
+        # Llama has no pad token by default
+        # https://github.com/huggingface/transformers/blob/07998ef39926b76d3f6667025535d0859eed61c3/docs/source/en/llm_tutorial.md?plain=1#L125
+        tokenizer.pad_token = tokenizer.eos_token
         model.config.eos_token_id = tokenizer.eos_token_id
         model.config.pad_token_id = tokenizer.pad_token_id
         return model, tokenizer


Test patch:
Repository: xorbitsai/inference

Issue:
FEAT: support Mistral-7B-Instruct
### Is your feature request related to a problem? Please describe
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1




Discussion:

Resolving patch:
diff --git a/xinference/model/llm/ggml/ctransformers.py b/xinference/model/llm/ggml/ctransformers.py
--- a/xinference/model/llm/ggml/ctransformers.py
+++ b/xinference/model/llm/ggml/ctransformers.py
@@ -190,7 +190,7 @@ def load(self):
     def match(
         cls, llm_family: LLMFamilyV1, llm_spec: LLMSpecV1, quantization: str
     ) -> bool:
-        if llm_spec.model_format != "ggmlv3":
+        if llm_spec.model_format != "ggmlv3" and llm_spec.model_format != "ggufv2":
             return False
         if llm_family.model_name not in CTRANSFORMERS_SUPPORTED_MODEL:
             return False


Test patch: