eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have some pascal-cased text that I'm trying to split into separate tokens/words. For example, "Hello123AIIsCool" would become ["Hello", "123", "AI", "Is", "Cool"]. Some Conditions  "Words" will always start with an upper-cased letter. E.g., "Hello" A contiguous sequence of numbers should be left together. E.g., "123" -> ["123"], not ["1", "2", "3"] A contiguous sequence of upper-cased letters should be kept together except when the last letter is the start to a new word as defined in the first condition. E.g., "ABCat" -> ["AB", "Cat"], not ["ABC", "at"] There is no guarantee that each condition will have a match in a string. E.g., "Hello", "HelloAI", "HelloAIIsCool" "Hello123", "123AI", "AIIsCool", and any other combination I haven't provided are potential candidates.  I've tried a couple regex variations. The following two attempts got me pretty close to what I want, but not quite. Version 0 import re def extract_v0(string: str) -> list[str]: word_pattern = r"[A-Z][a-z]*" num_pattern = r"\d+" pattern = f"{word_pattern}|{num_pattern}" extracts: list[str] = re.findall( pattern=pattern, string=string ) return extracts string = "Hello123AIIsCool" extract_v0(string)  ['Hello', '123', 'A', 'I', 'Is', 'Cool']  Version 1 import re def extract_v1(string: str) -> list[str]: word_pattern = r"[A-Z][a-z]+" num_pattern = r"\d+" upper_pattern = r"[A-Z][^a-z]*" pattern = f"{word_pattern}|{num_pattern}|{upper_pattern}" extracts: list[str] = re.findall( pattern=pattern, string=string ) return extracts string = "Hello123AIIsCool" extract_v1(string)  ['Hello', '123', 'AII', 'Cool']  Best Option So Far This uses a combination of regex and looping. It works, but is this the best solution? Or is there some fancy regex that can do it? import re def extract_v2(string: str) -> list[str]: word_pattern = r"[A-Z][a-z]+" num_pattern = r"\d+" upper_pattern = r"[A-Z][A-Z]*" groups = [] for pattern in [word_pattern, num_pattern, upper_pattern]: while string.strip(): group = re.search(pattern=pattern, string=string) if group is not None: groups.append(group) string = string[:group.start()] + " " + string[group.end():] else: break ordered = sorted(groups, key=lambda g: g.start()) return [grp.group() for grp in ordered] string = "Hello123AIIsCool" extract_v2(string)  ['Hello', '123', 'AI', 'Is', 'Cool'] 

Answer: Based on your Version 1: import re def extract_v1(string: str) -> list[str]: word_pattern = r"[A-Z][a-z]+" num_pattern = r"\d+" upper_pattern = r"[A-Z]+(?![a-z])" # Fixed pattern = f"{word_pattern}|{num_pattern}|{upper_pattern}" extracts: list[str] = re.findall( pattern=pattern, string=string ) return extracts string = "Hello123AIIsCool" extract_v1(string)  Result: ['Hello', '123', 'AI', 'Is', 'Cool']  The fixed upper_pattern will match as many uppercased letters as possible, and will stop one before a lowercased letter if it exists.
Question: Trying to install top2vec on colab ,and install everything that other people mentioned, but still get this error, have no idea how to solve,anybody knows? really appreciate! error screenshot !pip install top2vec !pip install top2vec[sentence_transformers] !pip install top2vec[sentence_encoders] from top2vec import Top2Vec import pandas as pd  ============================================================ AttributeError: module 'keras._tf_keras.keras' has no attribute 'internal'

Answer: This is a known issue: The recent release of Keras 3 breaks TensorFlow Probability at import. installation of tensorflow v2.15.0, tensorflow-probability v0.23.0, and keras v3 causes a AttributeError: module 'keras._tf_keras.keras' has no attribute '__internal__'   Please see these posts: https://github.com/tensorflow/probability/issues/1774#issuecomment-1979642276 First install tensorflow-probability version 0.24.0 then install tensorflow-keras  https://huggingface.co/google/gemma-7b-it/discussions/71
Question: To me, the only difference is that the regular operation needs one more instantiation, and the result is held by this new instance. And thus the regular implementation should call the other. But : these (in-place) methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). If a specific method is not defined, or if that method returns NotImplemented, the augmented assignment falls back to the normal methods. Here, i understand that the standard way is the opposite of mine : __iadd__ relies on __add__ EDIT in the sense that __add__ will be called in last resort. So why __add__ is the default way to achieve the addition and not __iadd__, whereas it should require less processing ? END EDITING A bit of context : the question came while implementing a Polynomial class, for learning purpose. I have written: class A: ... def __iadd__(self, other): "processing resulting in modification of attributes of self" return self def __add__(self, other): res = self.copy() # A.copy() being implemented as well res += other return res 

Answer: Inplace operations likely modify their operands. In the example you give, you use A.copy() to avoid that. Some types do not allow modifications (tuples, for example, are immutable). Thus they don't allow inplace operations. Also, inplace operations are not exactly intuitive to someone new to programming, or even to someone new to python. It's a nice feature of the language, but that's all. Overall, regular operations are the 'normal' form of operations. If you're writing a new class and want to support operators, you'll probably think of the regular operators first, and the inplace ones later, or not at all. With your way, inplace would not work if only regular was defined. But sometimes performance is indeed an issue, and inplace would allow for a more efficient implementation. That's why you have the possibility to define the inplace operation rather then just have it call the regular one.
Question: I have json data that is structured like this, which I want to turn into a data frame: { "data": { "1": { "Conversion": { "id": "1", "datetime": "2024-03-26 08:30:00" } }, "50": { "Conversion": { "id": "50", "datetime": "2024-03-27 09:00:00" } } } }  My usual approach would be to use json_normalize, like this: df = pd.json_normalize(input['data']) My goal is to have a table/dataframe with just the columns "id" and "datetime". How do I skip the numbering level below data and go straight to Conversion? I would imagine something like this (which clearly doesn't work): df = pd.json_normalize(input['data'][*]['Conversion']) What is the best way to achieve this? Any hints are greatly appreciated!

Answer: You have to manually change data in double list comprehension: L = [b['Conversion'] for k, v in input['data'].items() for a, b in v.items()] print (L) [{'id': '1', 'datetime': '2024-03-26 08:30:00'}, {'id': '50', 'datetime': '2024-03-27 09:00:00'}] out = pd.json_normalize(L) print (out) id datetime 0 1 2024-03-26 08:30:00 1 50 2024-03-27 09:00:00  Here is json_normalize not necessary, working DataFrame constructor: out = pd.DataFrame(L) print (out) id datetime 0 1 2024-03-26 08:30:00 1 50 2024-03-27 09:00:00  Thank you chepner for another idea with .values: out = pd.json_normalize((b['Conversion'] for v in input['data'].values() for b in v.values())) print (out) id datetime 0 1 2024-03-26 08:30:00 1 50 2024-03-27 09:00:00 out = pd.DataFrame((b['Conversion'] for v in input['data'].values() for b in v.values())) print (out) id datetime 0 1 2024-03-26 08:30:00 1 50 2024-03-27 09:00:00  In json_normalize is parameter max_level, but working different:  Max number of levels(depth of dict) to normalize. if None, normalizes all levels.  out = pd.json_normalize(input['data'], max_level=1) print (out) data.1 \ 0 {'Conversion': {'id': '1', 'datetime': '2024-0... data.50 0 {'Conversion': {'id': '50', 'datetime': '2024-... out = pd.json_normalize(input['data'], max_level=2) print (out) data.1.Conversion \ 0 {'id': '1', 'datetime': '2024-03-26 08:30:00'} data.50.Conversion 0 {'id': '50', 'datetime': '2024-03-27 09:00:00'} out = pd.json_normalize(input['data'], max_level=3) print (out) data.1.Conversion.id data.1.Conversion.datetime data.50.Conversion.id \ 0 1 2024-03-26 08:30:00 50 data.50.Conversion.datetime 0 2024-03-27 09:00:00
Question: df is a demo Polars DataFrame: df = pl.DataFrame( { "groups": ["A", "A", "A", "B", "B", "B"], "values": [1, 2, 3, 4, 5, 6], } )  The current group_by.agg() apporach is a bit inconvinient for creating descriptive statistics: print( df.group_by("groups").agg( pl.len().alias("count"), pl.col("values").mean().alias("mean"), pl.col("values").std().alias("std"), pl.col("values").min().alias("min"), pl.col("values").quantile(0.25).alias("25%"), pl.col("values").quantile(0.5).alias("50%"), pl.col("values").quantile(0.75).alias("75%"), pl.col("values").max().alias("max"), pl.col("values").skew().alias("skew"), pl.col("values").kurtosis().alias("kurtosis"), ) ) out: shape: (2, 11) ┌────────┬───────┬──────┬─────┬───┬─────┬─────┬──────┬──────────┐ │ groups ┆ count ┆ mean ┆ std ┆ … ┆ 75% ┆ max ┆ skew ┆ kurtosis │ │ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ u32 ┆ f64 ┆ f64 ┆ ┆ f64 ┆ i64 ┆ f64 ┆ f64 │ ╞════════╪═══════╪══════╪═════╪═══╪═════╪═════╪══════╪══════════╡ │ B ┆ 3 ┆ 5.0 ┆ 1.0 ┆ … ┆ 6.0 ┆ 6 ┆ 0.0 ┆ -1.5 │ │ A ┆ 3 ┆ 2.0 ┆ 1.0 ┆ … ┆ 3.0 ┆ 3 ┆ 0.0 ┆ -1.5 │ └────────┴───────┴──────┴─────┴───┴─────┴─────┴──────┴──────────┘  I want to write a customized group_by extension module that allows me to achieve the same results by calling: df.describe(by="groups", percentiles=[xxx], skew=True, kurt=True)  or df.group_by("groups").describe(percentiles=....) 

Answer: Calling this will output as same as you mentioned in the question import polars as pl class DescribeAccessor: def __init__(self, df: pl.DataFrame): self._df = df def __call__( self, by: str, percentiles: list = [0.25, 0.5, 0.75], skew: bool = True, kurt: bool = True, ) -> pl.DataFrame: percentile_exprs = [ pl.col("values").quantile(p).alias(f"{int(p * 100)}%") for p in percentiles ] aggs = [ pl.len().alias("count"), pl.col("values").mean().alias("mean"), pl.col("values").std().alias("std"), pl.col("values").min().alias("min"), *percentile_exprs, pl.col("values").max().alias("max"), ] if skew: aggs.append(pl.col("values").skew().alias("skew")) if kurt: aggs.append(pl.col("values").kurtosis().alias("kurtosis")) return self._df.group_by(by).agg(aggs) pl.DataFrame.describe = property(lambda self: DescribeAccessor(self)) df = pl.DataFrame( { "groups": ["A", "A", "A", "B", "B", "B"], "values": [1, 2, 3, 4, 5, 6], } ) print(df.describe(by="groups", percentiles=[0.25, 0.5, 0.75], skew=True, kurt=True))
Question: I want to build a RAG (Retrieval Augmented Generation) service with LangChain and for the retriever I want to use Solr. There is already a python package eurelis-langchain-solr-vectorstore where you can use Solr in combination with LangChain but how do I define server credentials? And my embedding model is already running on a server. I thought something like this but I don't know import requests from eurelis_langchain_solr_vectorstore import Solr embeddings_model = requests.post("http://server-insight/embeddings/") solr = Solr(embeddings_model, core_kwargs={ 'page_content_field': 'text_t', # field containing the text content 'vector_field': 'vector', # field containing the embeddings of the text content 'core_name': 'langchain', # core name 'url_base': 'http://localhost:8983/solr' # base url to access solr }) # with custom default core configuration retriever = solr.as_retriever() 

Answer: For the first question: For basic credentials you can send them in the url with the login:password@ pattern http://localhost:8983/solr => http://login:password@localhost:8983/solr For the second one: to use your embeddings server you need to provide the Solr vector store with a class inheriting from langchain_core.embeddings.embeddings.Embeddings it must then implement both def embed_documents(self, texts: List[str]) -> List[List[float]]: """Embed search docs."""  and def embed_query(self, text: str) -> List[float]: """Embed query text."""  in both methods you can use your http://server-insight/embeddings/ endpoint. First method is used at indexing time and is intended to work with a list of text and return a list of embeddings second one is used at query time and is intended to work with a single text and return a single embedding (a single list of float)
Question: I am trying to follow this article on medium Article. I had a few problems with it so the remain chang eI did was to the TrainingArguments object I added gradient_checkpointing_kwargs={'use_reentrant':False},. So now I have the following objects: peft_training_args = TrainingArguments( output_dir = output_dir, warmup_steps=1, per_device_train_batch_size=1, gradient_accumulation_steps=4, max_steps=100, #1000 learning_rate=2e-4, optim="paged_adamw_8bit", logging_steps=25, logging_dir="./logs", save_strategy="steps", save_steps=25, evaluation_strategy="steps", eval_steps=25, do_eval=True, gradient_checkpointing=True, gradient_checkpointing_kwargs={'use_reentrant':False}, report_to="none", overwrite_output_dir = 'True', group_by_length=True, ) peft_model.config.use_cache = False peft_trainer = transformers.Trainer( model=peft_model, train_dataset=train_dataset, eval_dataset=eval_dataset, args=peft_training_args, data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False), )  And when I call peft_trainer.train() I get the following error: AttributeError: 'torch.dtype' object has no attribute 'itemsize'  I'm using Databricks, and my pytorch version is 2.0.1+cu118

Answer: I was able to recreate your problem on Databricks with the following cluster:  Runtime: 14.1 ML (includes Apache Spark 3.5.0, GPU, Scala 2.12) Worker Type: Standard_NC16as_T4_v3 / Standard_NC6s_vs Driver Type: Standard_NC16as_T4_v3 / Standard_NC6s_vs  And then building on top of all the answers here already I was able to overcome your problem by the following:  Upgrade your transformers library via: !pip install -–upgrade git+https://github.com/huggingface/transformers Upgrading your torch version via: !pip install -–upgrade torch torchvision Upgrading your accelerate version via: !pip install -–upgrade accelerate Using a specific version of the datasets library via: !pip install datasets==2.16.0   I'm not sure if it matters but the order I used of the commands above are: 4 >> 1 >> 3 >> 2  This makes your problem go away and works on both transformers.Trainer and also SFTTrainer that I saw in your article imported but never used.
Question: I'm trying to reproduce the experiments of the S5 model, https://github.com/lindermanlab/S5, but I encountered some issues when solving the environment. When I'm running the shell script./run_lra_cifar.sh, I get the following error Traceback (most recent call last): File "/Path/S5/run_train.py", line 3, in <module> from s5.train import train File "/Path/S5/s5/train.py", line 7, in <module> from .train_helpers import create_train_state, reduce_lr_on_plateau,\ File "/Path/train_helpers.py", line 6, in <module> from flax.training import train_state File "/Path/miniconda3/lib/python3.12/site-packages/flax/__init__.py", line 19, in <module> from . import core File "/Path/miniconda3/lib/python3.12/site-packages/flax/core/__init__.py", line 15, in <module> from .axes_scan import broadcast File "/Path/miniconda3/lib/python3.12/site-packages/flax/core/axes_scan.py", line 22, in <module> from jax import linear_util as lu ImportError: cannot import name 'linear_util' from 'jax' (/Path/miniconda3/lib/python3.12/site-packages/jax/__init__.py)  I'm running this on an RTX4090 and my CUDA version is 11.8. My jax version is 0.4.25 and jaxlib version is 0.4.25+cuda11.cudnn86 I first tried to install the dependencies using the author's pip install -r requirements_gpu.txt  However, this doesn't seem to work in my case since I can't evenimport jax. So I installed jax according to the instructions on https://jax.readthedocs.io/en/latest/installation.html by typing pip install --upgrade pip pip install --upgrade "jax[cuda11_pip]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html  So far I've tried:  Using a older GPU(3060 and 2070) Downgrading python to 3.9  Does anyone know what could be wrong? Any help is appreciated

Answer: jax.linear_util was deprecated in JAX v0.4.16 and removed in JAX v0.4.24. It appears that flax is the source of the linear_util import, meaning that you are using an older flax version with a newer jax version. To fix your issue, you'll either need to install an older version of JAX which still has jax.linear_util, or update to a newer version of flax which is compatible with more recent JAX versions.
Question: I use a pyproject.toml file to list a package's dependencies: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "foobar" version = "1.0" requires-python = ">=3.8" dependencies = [ "requests>=2.0", "numpy", "tomli;python_version<'3.11'", ]  Is is possible, from within the package, to get the list of its own dependencies as strings? In the above case, it should give ["requests", "numpy"]  if used with Python>=3.11, and ["requests", "numpy", "tomli"]  otherwise.

Answer: Something along the lines of the following should do the trick: import importlib.metadata import packaging.requirements def _get_dependencies(name): rd = metadata(name).get_all('Requires-Dist') deps = [] for req in rd: req = packaging.requirements.Requirement(req) if req.marker is not None and not req.marker.evaluate(): continue deps.append(req.name) return deps  References:  https://docs.python.org/3/library/importlib.metadata.html#distribution-metadata https://importlib-metadata.readthedocs.io/en/latest/api.html#importlib_metadata.metadata https://packaging.pypa.io/en/stable/requirements.html
Question: I am trying to convert the following column into new rows:    Id Prices     001 ["March:59", "April:64", "May:62"]   002 ["Jan:55", ETC]    to    id date price     001 March 59   001 April 64   001 May 62   002 Jan 55    The date:price pairs aren't stored in a traditional dictionary format like the following solution: Convert dictionary keys to rows and show all the values in single column using Pandas I managed to get the key:value pairs into individual rows like:    Id Prices     001 March:59   001 April:64    And could split these into two columns using string manipulation but this feels inefficient instead of actually using the key:value pairs. Can anyone help please?

Answer: If you have valid lists, explode and split: df = pd.DataFrame({'Id': ['001', '002'], 'Prices': [["March:59", "April:64", "May:62"], ["Jan:55"]]}) out = df.explode('Prices') out[['date', 'price']] = out.pop('Prices').str.split(':', expand=True)  If you have strings, str.extractall with a regex and join: df = pd.DataFrame({'Id': ['001', '002'], 'Prices': ['["March:59", "April:64", "May:62"]', '["Jan:55"]']}) out = (df.drop(columns='Prices') .join(df['Prices'].str.extractall(r'(?P<date>[^":]+):(?P<price>[^":]+)') .droplevel('match')) )  Output:  Id date price 0 001 March 59 0 001 April 64 0 001 May 62 1 002 Jan 55  regex demo for the second approach.
Question: I have a LazyFrame of time, symbols and mid_price: Example: time symbols mid_price datetime[ns] str f64 2024-03-01 00:01:00 "PERP_SOL_USDT@… 126.1575 2024-03-01 00:01:00 "PERP_WAVES_USD… 2.71235 2024-03-01 00:01:00 "SOL_USDT@BINAN… 126.005 2024-03-01 00:01:00 "WAVES_USDT@BIN… 2.7085 2024-03-01 00:02:00 "PERP_SOL_USDT@… 126.3825  I want to perform some aggregations over the time dimension (ie: group by symbol): aggs = ( df .group_by('symbols') .agg([ pl.col('mid_price').diff(1).alias("change"), ]) )  I get back a list of each value per unique symbols value: >>> aggs.head().collect() symbols change str list[f64] "SOL_USDT@BINAN… [null, 0.25, … -0.55] "PERP_SOL_USDT@… [null, 0.225, … -0.605] "WAVES_USDT@BIN… [null, -0.002, … -0.001] "PERP_WAVES_USD… [null, -0.00255, … 0.0001]  I would now like to join this back onto my original dataframe: df = df.join( aggs, on='symbols', how='left', )  This now results in each row getting the full list of change, rather then the respective value. >>> df.head().collect() time symbols mid_price change datetime[ns] str f64 list[f64] 2024-03-01 00:01:00 "PERP_SOL_USDT@… 126.1575 [null, 0.225, … -0.605] 2024-03-01 00:01:00 "PERP_WAVES_USD… 2.71235 [null, -0.00255, … 0.0001] 2024-03-01 00:01:00 "SOL_USDT@BINAN… 126.005 [null, 0.25, … -0.55] 2024-03-01 00:01:00 "WAVES_USDT@BIN… 2.7085 [null, -0.002, … -0.001] 2024-03-01 00:02:00 "PERP_SOL_USDT@… 126.3825 [null, 0.225, … -0.605]  I have 2 questions please:  How do I unstack/explode the lists returned from my group_by when joining them back into the original dataframe? Is this the recommended way to add a new column to my original dataframe from a group_by (that is: group_by followed by join)? 

Answer: It sounds like you don't want to actually aggregate anything (and get a single value per symbol), but instead want to compute "change" but independently for each symbol. In polars, this kind of behaviour, similar to window functions in PostgreSQL, can be achieved with pl.Expr.over. df.with_columns( pl.col("mid_price").diff(1).over("symbol").alias("change") )  On some example data, the resolt looks as follows. import polars as pl import numpy as np import datetime df = pl.DataFrame({ "symbol": ["A"] * 3 + ["B"] * 3 + ["C"] * 3, "time": [datetime.datetime(2024, 3, 1, hour) for hour in range(3)] * 3, "mid_price": np.random.randn(9), }) df.with_columns( pl.col("mid_price").diff(1).over("symbol").alias("change") )  shape: (9, 4) ┌────────┬─────────────────────┬───────────┬───────────┐ │ symbol ┆ time ┆ mid_price ┆ change │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ datetime[μs] ┆ f64 ┆ f64 │ ╞════════╪═════════════════════╪═══════════╪═══════════╡ │ A ┆ 2024-03-01 00:00:00 ┆ -0.349863 ┆ null │ │ A ┆ 2024-03-01 01:00:00 ┆ 0.093732 ┆ 0.443595 │ │ A ┆ 2024-03-01 02:00:00 ┆ -1.262064 ┆ -1.355796 │ │ B ┆ 2024-03-01 00:00:00 ┆ 1.953929 ┆ null │ │ B ┆ 2024-03-01 01:00:00 ┆ 0.637582 ┆ -1.316348 │ │ B ┆ 2024-03-01 02:00:00 ┆ 1.009401 ┆ 0.37182 │ │ C ┆ 2024-03-01 00:00:00 ┆ 0.75864 ┆ null │ │ C ┆ 2024-03-01 01:00:00 ┆ -0.866227 ┆ -1.624867 │ │ C ┆ 2024-03-01 02:00:00 ┆ -0.674938 ┆ 0.191289 │ └────────┴─────────────────────┴───────────┴───────────┘
Question: Timing results in Python 3.12 (and similar with 3.11 and 3.13 on different machines): When x = None: 13.8 ns x is None 10.1 ns if x is None: pass When x = True: 13.9 ns x is None 11.1 ns if x is None: pass  How can doing more take less time? Why is if x is None: pass faster, when it does the same x is None check and then additionally checks the truth value of the result (and does or skips the pass)? Times on other versions/machines:  Python 3.11: (12.4, 9.3) and (12.0, 8.8) Python 3.13: (12.7, 9.9) and (12.7, 9.6)  Benchmark script (Attempt This Online!): from timeit import repeat import sys for x in None, True: print(f'When {x = }:') for code in ['x is None', 'if x is None: pass'] * 2: t = min(repeat(code, f'{x=}', repeat=100)) print(f'{t*1e3:4.1f} ns ', code) print() print('Python:', sys.version) 

Answer: Look at the disassembled code: >>> import dis >>> dis.dis('if x is None: pass') 0 0 RESUME 0 1 2 LOAD_NAME 0 (x) 4 POP_JUMP_IF_NOT_NONE 1 (to 8) 6 RETURN_CONST 0 (None) >> 8 RETURN_CONST 0 (None) >>> dis.dis('x is None') 0 0 RESUME 0 1 2 LOAD_NAME 0 (x) 4 LOAD_CONST 0 (None) 6 IS_OP 0 8 RETURN_VALUE  The if case has a special POP_JUMP_IF_NOT_NONE operation, which is faster than a LOAD_CONST plus IS_OP. You can read the detailed discussion about it here: https://github.com/faster-cpython/ideas/discussions/154.