eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to write a function, that will find the first occurrence of two adjacent characters that are the same, replace them with a single character that is next in the alphabet and go over the string until there are no duplicates left. In case of "zz" it should go in a circular fashion back to "a". The string can only include characters a-z, that is, no capital letters or non-alphabetical characters. I have written a function that does it, but it is not effective enough for a very long string. def solve(s): i = 1 while i < len(s): if s[i] == s[i-1]: r = s[i+1:] l = s[:i-1] if s[i] == "z": x = "a" else: x = chr(ord(s[i])+1) i = 1 s = l+x+r else: i += 1 return s  So for example if s = 'aabbbc' the function should work like aabbbc --> bbbbc --> cbbc --> ccc and finally return dc. How can I make it more efficient? Edit: for example if s = 'ab'*10**4 + 'cc'*10**4 + 'dd'*10**4 this function is taking a lot of time.

Answer: As a trivial optimisation: instead of the hard reset i = 1, you can use a softer reset i = i-1. Indeed, if there was no duplicate between 1 and i before the transformation, then there still won't be any duplicate between 1 and i-1 after the transformation. def solve(s): i = 1 while i < len(s): if s[i] == s[i-1]: l = s[:i-1] r = s[i+1:] # I swapped these two lines because I read left-to-right if s[i] == "z": x = "a" else: x = chr(ord(s[i])+1) s = l+x+r i = max(1, i-1) # important change is here else: i += 1 return s s = 'ab'*10**4 + 'cc'*10**4 + 'dd'*10**4 t = solve(s) print(t[2*10**4:]) # rqpnlih print(t == 'ab'*10**4 + 'rqpnlih') # True
Question: I have polars dataframe with timestamp folumn of type datetime[ns] which value is 2023-03-08 11:13:07.831 I want to use polars efficiency to round timestamp to 5 minutes floor. Right now I do: import arrow def timestamp_5minutes_floor(ts: int) -> int: return int(arrow.get(ts).timestamp() // 300000 * 300000) df.with_columns([ pl.col("timestamp").apply(lambda x: timestamp_5minutes_floor(x)).alias("ts_floor") ])  It is slow. How to improve it?

Answer: You could try to use .dt.truncate: With the sample dataframe df = pl.DataFrame({ "ts": ["2023-03-08 11:01:07.831", "2023-03-08 18:09:01.007"] }).select(pl.col("ts").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S%.3f"))  ┌─────────────────────────┐ │ ts │ │ --- │ │ datetime[ms] │ ╞═════════════════════════╡ │ 2023-03-08 11:01:07.831 │ │ 2023-03-08 18:09:01.007 │ └─────────────────────────┘  this df = df.select(pl.col("ts").dt.truncate("5m"))  results in ┌─────────────────────┐ │ ts │ │ --- │ │ datetime[ms] │ ╞═════════════════════╡ │ 2023-03-08 11:00:00 │ │ 2023-03-08 18:05:00 │ └─────────────────────┘
Question: I have a reasonably large dataframe on hand. Joining it with itself takes some time. But I want to join them with some conditions, which could make the resulting dataframe much smaller. My question is how can I take advantage of such conditions to make the conditional join faster to plain full join? Code below for illustration: import time import numpy as np import polars as pl # example dataframe rng = np.random.default_rng(1) nrows = 3_000_000 df = pl.DataFrame( dict( day=rng.integers(1, 300, nrows), id=rng.integers(1, 5_000, nrows), id2=rng.integers(1, 5, nrows), value=rng.normal(0, 1, nrows), ) ) # joining df with itself takes around 10-15 seconds on a machine with 32 cores. start = time.perf_counter() df.join(df, on=["id", "id2"], how="left") time.perf_counter() - start # joining df with itself with extra conditions - the implementation below that takes very similar time (10-15 seconds). start = time.perf_counter() df.join(df, on=["id", "id2"], how="left").filter( (pl.col("day") < pl.col("day_right")) & (pl.col("day_right") - pl.col("day") <= 30) ) time.perf_counter() - start  So, as mentioned above, my question is how can I take advantage of the conditions during join to make the 'conditional join' faster? It should be faster since the resulting dataframe after conditions has 10x less rows than the full join without any conditions.

Answer: With lazy and streaming=True, it is faster: In [5]: start = time.perf_counter() ...: df.lazy().join(df.lazy(), on=["id", "id2"], how="left").filter( ...: (pl.col("day") < pl.col("day_right")) & (pl.col("day_right") - pl.col("day") <= 30) ...: ).collect() ...: time.perf_counter() - start Out[5]: 11.083821532000002 In [6]: start = time.perf_counter() ...: df.lazy().join(df.lazy(), on=["id", "id2"], how="left").filter( ...: (pl.col("day") < pl.col("day_right")) & (pl.col("day_right") - pl.col("day") <= 30) ...: ).collect(streaming=True) ...: time.perf_counter() - start Out[6]: 7.110704054997768
Question: I want to parse TOML files in python 3.9, and I am wondering if I can do so without installing another package. Since pip knows how to work with pyproject.toml files, and I already have pip installed, does pip provide a parser that I can import and use in my own code?

Answer: For 3.9, pip vendors tomli: from pip._vendor import tomli  For consenting adults, importing the vendored module should work as usual. However, this is an implementation detail of pip, and it could change without a deprecation period at any moment in a future release of pip. Therefore, for anything apart from a quick hack, it would be safer to install tomli (or some other TOML parser) into the site instead.
Question: I'm trying to train a LGBClassifier for multiclass task. I tried first working directly with LightGBM API and set the model and training as follows: LightGBM API train_data = lgb.Dataset(X_train, (y_train-1)) test_data = lgb.Dataset(X_test, (y_test-1)) params = {} params['learning_rate'] = 0.3 params['boosting_type'] = 'gbdt' params['objective'] = 'multiclass' params['metric'] = 'softmax' params['max_depth'] = 10 params['num_class'] = 8 params['num_leaves'] = 500 lgb_train = lgb.train(params, train_data, 200) # AFTER TRAINING THE MODEL y_pred = lgb_train.predict(X_test) y_pred_class = [np.argmax(line) for line in y_pred] y_pred_class = np.asarray(y_pred_class) + 1  This is how the confussion matrix looks:  Sklearn API Then I tried to move to Sklearn API to be able to use other tools. This is the code I used: lgb_clf = LGBMClassifier(objective='multiclass', boosting_type='gbdt', max_depth=10, num_leaves=500, learning_rate=0.3, eval_metric=['accuracy','softmax'], num_class=8, n_jobs=-1, early_stopping_rounds=100, num_iterations=500) clf_train = lgb_clf(X_train, (y_train-1), verbose=1, eval_set=[(X_train, (y_train-1)), (X_test, (y_test-1)))]) # TRAINING: I can see overfitting is happening y_pred = clf_train.predict(X_test) y_pred = [np.argmax(line) for line in y_pred] y_pred = np.asarray(y_pred) + 1  And this is the confusion matrix in this case:  Notes  I need to substract 1 from y_train as my classes start at 1 and LightGBM was complaining about this. When I try a RandomSearch or a GridSearch I always obtain the same result as the last confusion matrix. I have check different questions here but none solve this issue.  Questions  Is there anything that I'm missing out when implementing the model in Sklearn API? Why do I obtain good results (maybe with overfitting) with LightGBM API? How can I achieve the same results with the two APIs?  Thanks in advance. UPDATE It was my mistake. I thought the output in both APIs would be the same but it doesn't seem like that. I just removed the np.argmax() line when predicting with Sklearn API. It seems this API already predict directly the class. Don't remove the question in case someone else is dealing with similar issues.

Answer: It was my mistake. I thought the output in both APIs would be the same but it doesn't seem like that. I just removed the np.argmax() line when predicting with Sklearn API. It seems this API already predict directly the class. Don't remove the question in case someone else is dealing with similar issues.
Question: Currently, using the polars' cast() method on columns of type list[] is not supported. It throws: ComputeError: Cannot cast list type  Before I do as usual (use rows(), or convert to pandas, or work with apply()). Is there any trick or best practice to convert polars list[] to strings? Here a quick snippet for you to replicate the error df = pl.from_dict({'foo': [[1,2,3]], 'bar': 'Hello World'}) print(df) ''' shape: (1, 2) ┌───────────┬─────────────┐ │ foo ┆ bar │ │ --- ┆ --- │ │ list[i64] ┆ str │ ╞═══════════╪═════════════╡ │ [1, 2, 3] ┆ Hello World │ └───────────┴─────────────┘ ''' df['foo'].cast(str) # this other workaround wont work neither df.select([pl.col('foo').str])  Here is what i expect to see: ''' shape: (1, 2) ┌───────────┬─────────────┐ │ foo ┆ bar │ │ --- ┆ --- │ │ str ┆ str │ ╞═══════════╪═════════════╡ │"[1, 2, 3]"┆ Hello World │ └───────────┴─────────────┘ ''' 

Answer: You can use the datatype pl.List(pl.String) df.with_columns(pl.col("foo").cast(pl.List(pl.String)))  shape: (1, 2) ┌─────────────────┬─────────────┐ │ foo | bar │ │ --- | --- │ │ list[str] | str │ ╞═════════════════╪═════════════╡ │ ["1", "2", "3"] | Hello World │ └─────────────────┴─────────────┘  To create an actual string - perhaps: df.with_columns("[" + pl.col("foo").cast(pl.List(pl.String)).list.join(", ") + "]" )  There's also pl.format() df.with_columns( pl.format("[{}]", pl.col("foo").cast(pl.List(pl.String)).list.join(", ")))  shape: (1, 3) ┌───────────┬─────────────┬───────────┐ │ foo | bar | literal │ │ --- | --- | --- │ │ list[i64] | str | str │ ╞═══════════╪═════════════╪═══════════╡ │ [1, 2, 3] | Hello World | [1, 2, 3] │ └───────────┴─────────────┴───────────┘
Question: I have some duration type data (lap times) as pl.String that fails to convert using strptime, whereas regular datetimes work as expected. Minutes (before :) and Seconds (before .) are always padded to two digits, Milliseconds are always 3 digits. Lap times are always < 2 min. df = pl.DataFrame({ "lap_time": ["01:14.007", "00:53.040", "01:00.123"] }) df = df.with_columns( # pl.col('release_date').str.to_date("%B %d, %Y"), # works pl.col('lap_time').str.to_time("%M:%S.%3f").cast(pl.Duration), # fails )  So I used the chrono format specifier definitions from https://docs.rs/chrono/latest/chrono/format/strftime/index.html which are used as per the polars docs of strptime the second conversion (for lap_time) always fails, no matter whether I use .%f, .%3f, %.3f. Apparently, strptime doesn't allow creating a pl.Duration directly, so I tried with pl.Time but it fails with error: ComputeError: strict conversion to dates failed, maybe set strict=False  but setting strict=False yields all null values for the whole Series. Am I missing something or this some weird behavior on chrono's or python-polars part?

Answer: General case In case you have duration that may exceed 24 hours, you can extract data (minutes, seconds and so on) from string using regex pattern. For example: df = pl.DataFrame({ "time": ["+01:14.007", "100:20.000", "-05:00.000"] }) df.with_columns( pl.col("time").str.extract_all(r"([+-]?\d+)") # / # you will get array of length 3 # ["min", "sec", "ms"] ).with_columns( pl.duration( minutes=pl.col("time").list.get(0), seconds=pl.col("time").list.get(1), milliseconds=pl.col("time").list.get(2) ).alias("time") )  ┌──────────────┐ │ time │ │ --- │ │ duration[ns] │ ╞══════════════╡ │ 1m 14s 7ms │ │ 1h 40m 20s │ │ -5m │ └──────────────┘   About pl.Time To convert data to pl.Time, you need to specify hours as well. When you add 00 hours to your time, code will work: df = pl.DataFrame({"str_time": ["01:14.007", "01:18.880"]}) df.with_columns( duration = (pl.lit("00:") + pl.col("str_time")) .str.to_time("%T%.3f") .cast(pl.Duration) )  ┌───────────┬──────────────┐ │ str_time ┆ duration │ │ --- ┆ --- │ │ str ┆ duration[μs] │ ╞═══════════╪══════════════╡ │ 01:14.007 ┆ 1m 14s 7ms │ │ 01:18.880 ┆ 1m 18s 880ms │ └───────────┴──────────────┘
Question: I am new to Polars and I am not sure whether I am using .with_columns() correctly. Here's a situation I encounter frequently: There's a dataframe and in .with_columns(), I apply some operation to a column. For example, I convert some dates from str to date type and then want to compute the duration between start and end date. I'd implement this as follows. import polars as pl pl.DataFrame( { "start": ["01.01.2019", "01.01.2020"], "end": ["11.01.2019", "01.05.2020"], } ).with_columns( pl.col("start").str.to_date(), pl.col("end").str.to_date(), ).with_columns( (pl.col("end") - pl.col("start")).alias("duration"), )  First, I convert the two columns, next I call .with_columns() again. Something shorter like this does not work: pl.DataFrame( { "start": ["01.01.2019", "01.01.2020"], "end": ["11.01.2019", "01.05.2020"], } ).with_columns( pl.col("start").str.to_date(), pl.col("end").str.to_date(), (pl.col("end") - pl.col("start")).alias("duration"), )  # InvalidOperationError: sub operation not supported for dtypes `str` and `str`  Is there a way to avoid calling .with_columns() twice and to write this in a more compact way?

Answer: The second .with_columns() is needed. From the GitHub Issues  I don't want this extra complexity in polars. If you want to use an updated column, you need two with_columns. This makes it much more readable, simple, and explainable.  In the given example, passing multiple names to col() could simplify it slightly. (df.with_columns(pl.col("start", "end").str.to_date()) .with_columns(duration = pl.col("end") - pl.col("start")) )  shape: (2, 3) ┌────────────┬────────────┬──────────────┐ │ start ┆ end ┆ duration │ │ --- ┆ --- ┆ --- │ │ date ┆ date ┆ duration[ms] │ ╞════════════╪════════════╪══════════════╡ │ 2019-01-01 ┆ 2019-01-11 ┆ 10d │ │ 2020-01-01 ┆ 2020-05-01 ┆ 121d │ └────────────┴────────────┴──────────────┘
Question: Tried to create a random image with PIL as per the example: import numpy from PIL import image a = numpy.random.rand(48,84) img = Image.fromarray(a.astype('uint8')).convert('1') print(len(img.tobytes()))  This particular code will output 528. Wen we flip the numbers of the numpy array: a = numpy.random.rand(84,48)  The output we get is 504. Why is that? I was expecting for the byte number to be the same, since the numpy arrays are the same size.

Answer: When you call tobytes() on the boolean array*, the data is likely encoded per row. In your second example, there are 48 booleans in each row of img. So each row can be represented with 6 bytes (48 bits). 6 bytes * 84 rows = 504 bytes in img. However, in your first example, there are 84 pixels per row, which is not divisible by 8. In this case, the encoder represents each row with 11 bytes (88 bits). There are 4 extra bits of padding per row. So now the total size is 11 bytes * 48 rows = 528 bytes. If you test a bunch of random input shapes for a 2d boolean array to encode, you will find that when the number of elements per row is divisible by 8, the number of total bytes in the encoding is equal to the width * height / 8. However, when the row length is not divisible by 8, the encoding will contain more bytes because it has to pad each row with between 1 and 7 bits. In summary - ideally, we would want to store eight boolean values per byte, but this is complicated by the fact that the row length isn't always divisible by 8, and the encoder serializes the array by row. Edit for clarification: *the PIL.Image object in mode "1" (binary or "bilevel" image) effectively represents a boolean array. In mode 1, the original image (in this case, the numpy array a) is thresholded to convert it to a binary image.
Question: I am trying to load this semantic segmentation model from HF using the following code: from transformers import pipeline model = pipeline("image-segmentation", model="Carve/u2net-universal", device="cpu")  But I get the following error: OSError: tamnvcc/isnet-general-use does not appear to have a file named config.json. Checkout 'https://huggingface.co/tamnvcc/isnet-general-use/main' for available files.  Is it even possible to load models from HuggingFace without config.json file provided? I also tried loading the model via: id2label = {0: "background", 1: "target"} label2id = {"background": 0, "target": 1} image_processor = AutoImageProcessor.from_pretrained("Carve/u2net-universal") model = AutoModelForSemanticSegmentation("Carve/u2net-universal", id2label=id2label, label2id=label2id)  But got the same error.

Answer: TL;DR You will need to make a lot of assumption if you don't have the config.json and the model card doesn't have any documentation After some guessing, possibly it's this: from u2net import U2NET import torch model = U2NET() model.load_state_dict(torch.load('full_weights.pth', map_location=torch.device('cpu')))   In Long Looking at the files available in the model card, we see these files:  .gitattributes README.md full_weights.pth  A good guess would be that the .pth file is a PyTorch model binary. Given that, we can try: import shutil import requests import torch # Download the .pth file locally url = "https://huggingface.co/Carve/u2net-universal/resolve/main/full_weights.pth" response = requests.get(url, stream=True) with open('full_weights.pth', 'wb') as out_file: shutil.copyfileobj(response.raw, out_file) model = torch.load('full_weights.pth', map_location=torch.device('cpu'))  But what you end up with is NOT a usable model, it's just the model parameters/weights (aka checkpoint file), i.e. type(model)  [out]: collections.OrderedDict  Looking at the layer names, it looks like a rebnconvin model that points to the https://github.com/xuebinqin/U-2-Net code: model.keys()  [out]: odict_keys(['stage1.rebnconvin.conv_s1.weight', 'stage1.rebnconvin.conv_s1.bias', 'stage1.rebnconvin.bn_s1.weight', 'stage1.rebnconvin.bn_s1.bias', 'stage1.rebnconvin.bn_s1.running_mean', 'stage1.rebnconvin.bn_s1.running_var', 'stage1.rebnconv1.conv_s1.weight', 'stage1.rebnconv1.conv_s1.bias', 'stage1.rebnconv1.bn_s1.weight', 'stage1.rebnconv1.bn_s1.bias', 'stage1.rebnconv1.bn_s1.running_mean', 'stage1.rebnconv1.bn_s1.running_var', ...])  ASSUMING THAT YOU CAN TRUST THE CODE from the github, you can try installing it with: ! wget https://raw.githubusercontent.com/xuebinqin/U-2-Net/master/model/u2net.py  And guessing from the layer names and model name, it looks like a U2Net from https://arxiv.org/abs/2005.09007v3 So you can try: from u2net import U2NET model = U2NET() model.load_state_dict(torch.load('full_weights.pth', map_location=torch.device('cpu')))
Question: I've a set of test files (.py files) for different UI tests. I want to run these test files using pytest in a specific order. I used the below command python -m pytest -vv -s --capture=tee-sys --html=report.html --self-contained-html ./Tests/test_transTypes.py ./Tests/test_agentBank.py ./Tests/test_bankacct.py  The pytest execution is triggered from an AWS Batch job. When the test executions happens it is not executing the test files in the order as specified in the above command. Instead it first runs test_agentBank.py followed by test_bankacct.py, then test_transTypes.py Each of these python files contains bunch of test functions. I also tried decorating the test function class such as @pytest.mark.run(order=1) in the first python file(test_transTypes.py), @pytest.mark.run(order=2) in the 2nd python file(test_agentBank.py) etc. This seems to run the test in the order, but at the end I get a warning  PytestUnknownMarkWarning: Unknown pytest.mark.run - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs .pytest.org/en/stable/how-to/mark.html @pytest.mark.run(order=1)  What is the correct way of running tests in a specific order in pytest? Each of my "test_" python files are the ones I need to run using pytest. Any help much appreciated.

Answer: To specify the order in which tests are run in pytest, you can use the pytest-order plugin. This plugin allows you to customize the order in which your tests are run by providing the order marker, which has attributes that define when your tests should run in relation to each other. You can use absolute attributes (e.g., first, second-to-last) or relative attributes (e.g., run this test before this other test) to specify the order of your tests. Here's an example: import pytest @pytest.mark.order(2) def test_foo(): assert True @pytest.mark.order(1) def test_bar(): assert True
Question: I tried the following code but got only partial results like [{"light_id": 0, "color  I was expecting the full JSON as suggested on this page: https://medium.com/@richardhayes777/using-chatgpt-to-control-hue-lights-37729959d94f import json import os import time from json import JSONDecodeError from typing import List import openai openai.api_key = "xxx" HEADER = """ I have a hue scale from 0 to 65535. red is 0.0 orange is 7281 yellow is 14563 purple is 50971 pink is 54612 green is 23665 blue is 43690 Saturation is from 0 to 254 Brightness is from 0 to 254 Two JSONs should be returned in a list. Each JSON should contain a color and a light_id. The light ids are 0 and 1. The color relates a key "color" to a dictionary with the keys "hue", "saturation" and "brightness". Give me a list of JSONs to configure the lights in response to the instructions below. Give only the JSON and no additional characters. Do not attempt to complete the instruction that I give. Only give one JSON for each light. """ completion = openai.Completion.create(model="text-davinci-003", prompt=HEADER) print(completion.choices[0].text) 

Answer: In general GPT-3 API (i.e., Completions API) If you get partial completion (i.e., if the completion is cut off), it's because the max_tokens parameter is set too low or you didn't set it at all (in this case, it defaults to 16). You need to set it higher, but the token count of your prompt and completion together cannot exceed the model's context length. See the official OpenAI documentation:  GPT-3.5 and GPT-4 API (i.e., Chat Completions API) Compared to the GPT-3 API, the GPT-3.5 and GPT-4 APIs have the max_tokens parameter set to infinite by default.   Your case You're using text-davinci-003 (i.e., the GPT-3 API). If you don't set max_tokens = 1024 the completion you get will be cut off. Take a careful look at the tutorial you're referring to once again. If you run test.py, the OpenAI API will return a completion:  Light 0 should be red: [{"light_id": 0, "color": {"hue": 0, "saturation": 254, "brightness": 254}},{"light_id": 1, "color": }] Light 1 should be orange: [{"light_id": 0, "color": {"hue": 0, "saturation": 254, "brightness": 254}},{"light_id": 1, "color": {"hue": 7281, "saturation": 254, "brightness": 254}}]  test.py import openai import os openai.api_key = os.getenv('OPENAI_API_KEY') HEADER = """ I have a hue scale from 0 to 65535. red is 0.0 orange is 7281 yellow is 14563 purple is 50971 pink is 54612 green is 23665 blue is 43690 Saturation is from 0 to 254 Brightness is from 0 to 254 Two JSONs should be returned in a list. Each JSON should contain a color and a light_id. The light ids are 0 and 1. The color relates a key "color" to a dictionary with the keys "hue", "saturation" and "brightness". Give me a list of JSONs to configure the lights in response to the instructions below. Give only the JSON and no additional characters. Do not attempt to complete the instruction that I give. Only give one JSON for each light. """ completion = openai.Completion.create(model="text-davinci-003", prompt=HEADER, max_tokens=1024) print(completion.choices[0].text)