eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: import polars as pl df = pl.DataFrame({ 'A': range(1,4), 'B': range(1,4), 'C': range(1,4), 'D': range(1,4) }) print(df)  shape: (3, 4) ┌─────┬─────┬─────┬─────┐ │ A ┆ B ┆ C ┆ D │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╪═════╡ │ 1 ┆ 1 ┆ 1 ┆ 1 │ │ 2 ┆ 2 ┆ 2 ┆ 2 │ │ 3 ┆ 3 ┆ 3 ┆ 3 │ └─────┴─────┴─────┴─────┘  df_long = df.unpivot( variable_name="recipe", value_name="revenue") print(df_long)  shape: (12, 2) ┌────────┬─────────┐ │ recipe ┆ revenue │ │ --- ┆ --- │ │ str ┆ i64 │ ╞════════╪═════════╡ │ A ┆ 1 │ │ A ┆ 2 │ │ A ┆ 3 │ │ B ┆ 1 │ │ B ┆ 2 │ │ … ┆ … │ │ C ┆ 2 │ │ C ┆ 3 │ │ D ┆ 1 │ │ D ┆ 2 │ │ D ┆ 3 │ └────────┴─────────┘  It seems I need to add an index in order to pivot df_long back into the original form of df? Is there no way to pivot a polars dataframe without adding an index? df_long = df_long.with_columns(index=pl.col("revenue").cum_count().over("recipe")) df_long.pivot( on='recipe', index='index', values='revenue', aggregate_function='first' )  shape: (3, 5) ┌───────┬─────┬─────┬─────┬─────┐ │ index ┆ A ┆ B ┆ C ┆ D │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ u32 ┆ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═══════╪═════╪═════╪═════╪═════╡ │ 1 ┆ 1 ┆ 1 ┆ 1 ┆ 1 │ │ 2 ┆ 2 ┆ 2 ┆ 2 ┆ 2 │ │ 3 ┆ 3 ┆ 3 ┆ 3 ┆ 3 │ └───────┴─────┴─────┴─────┴─────┘  In R, I can perform the equivalent to unpivot and pivot without indexing, and was seeking the same functionality in Python. df_pandas = df.to_pandas()  library(tidyverse) library(reticulate) df_long <- py$df_pandas |> pivot_longer( everything(), names_to = 'recipe', values_to = 'value' ) df_long |> pivot_wider( names_from='recipe', values_from='value' ) |> unnest(cols = c(A,B,C,D)) 

Answer:  It seems I need to add an index in order to pivot df_long back into the original form of df?  How would it otherwise decide that the entries in df_long with revenue=1 are all on the same row, and not for recipe=A the revenue is 2, for recipe=B, the revenue is 3, etc? You are inferring that from how you defined df, but from just df_long as defined in your question, that is not possible. If this is a requirement, than I suggest you add the index to df before using unpivot, so that df_long carries that information: >>> df_long=df.with_row_index().unpivot(index=["index"], variable_name="recipe",value_name="revenue") >>> df_long shape: (12, 3) ┌───────┬────────┬─────────┐ │ index ┆ recipe ┆ revenue │ │ --- ┆ --- ┆ --- │ │ u32 ┆ str ┆ i64 │ ╞═══════╪════════╪═════════╡ │ 0 ┆ A ┆ 1 │ │ 1 ┆ A ┆ 2 │ │ 2 ┆ A ┆ 3 │ │ 0 ┆ B ┆ 1 │ │ 1 ┆ B ┆ 2 │ │ … ┆ … ┆ … │ │ 1 ┆ C ┆ 2 │ │ 2 ┆ C ┆ 3 │ │ 0 ┆ D ┆ 1 │ │ 1 ┆ D ┆ 2 │ │ 2 ┆ D ┆ 3 │ └───────┴────────┴─────────┘  The pivot can then use the index column to reconstruct the original.
Question: Below Python program is intended to translate large English texts into French. I use a for loop to feed a series of reports into Ollama. from functools import cached_property from ollama import Client class TestOllama: @cached_property def ollama_client(self) -> Client: return Client(host=f"http://127.0.0.1:11434") def translate(self, text_to_translate: str): ollama_response = self.ollama_client.generate( model="mistral", prompt=f"translate this French text into English: {text_to_translate}" ) return ollama_response['response'].lstrip(), ollama_response['total_duration'] def run(self): reports = ["reports_text_1", "reports_text_2"....] # avearge text size per report is between 750-1000 tokens. for each_report in reports: try: translated_report, total_duration = self.translate( text_to_translate=each_report ) print(f"Translated text:{translated_report}, Time taken:{total_duration}") except Exception as e: pass if __name__ == '__main__': job = TestOllama() job.run()  docker command to run ollama: docker run -d --gpus=all --network=host --security-opt seccomp=unconfined -v report_translation_ollama:/root/.ollama --name ollama ollama/ollama  My question is: When I run this script on V100 and H100, I don't see a significant difference in execution time. I've avoided parallelism, thinking that Ollama might internally use parallelism to process. However, when I check with the htop command, I see only one core being used. Am I correct in my understanding? I am a beginner in NLP, so any help or guidance on how to organize my code (e.g., using multithreading to send Ollama requests) would be appreciated.

Answer: Flags OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS were added in v0.1.33. You can set them when starting the Ollama server: OLLAMA_NUM_PARALLEL=4 OLLAMA_MAX_LOADED_MODELS=4 ollama serve  Available server settings  OLLAMA_MAX_LOADED_MODELS - The maximum number of models that can be loaded concurrently provided they fit in available memory. The default is 3 * the number of GPUs or 3 for CPU inference. OLLAMA_NUM_PARALLEL - The maximum number of parallel requests each model will process at the same time. The default will auto-select either 4 or 1 based on available memory. OLLAMA_MAX_QUEUE - The maximum number of requests Ollama will queue when busy before rejecting additional requests. The default is 512.  More information  Concurrency is no longer experimental as of v0.2.0 and newer. Leave these variables unset so the default algorithm is used. If you want more (or less) parallelism per model than 4 (the default) then you can adjust OLLAMA_NUM_PARALLEL. The default max loaded models is 3x GPU count. According to performance testing run by the Ollama maintainers, setting >3 often leads to degraded performance if all the models are being accessed concurrently, so make sure to run your own tests before settling on an alternative max models setting.  Source: faq.md#how-does-ollama-handle-concurrent-requests
Question: I have a button on a webpage which looks like this: <button rpl="" aria-controls="comment-children" aria-expanded="true" aria-label="Toggle Comment Thread" class="text-neutral-content-strong bg-neutral-background overflow-visible w-md h-md button-small px-[var(--rem6)] button-plain icon items-center justify-center button inline-flex "> <!--?lit$747127195$--><!----><span class="flex items-center justify-center"> <!--?lit$747127195$--><span class="flex"><!--?lit$747127195$--><svg rpl="" fill="currentColor" height="16" icon-name="leave-outline" viewBox="0 0 20 20" width="16" xmlns="http://www.w3.org/2000/svg"> <!--?lit$747127195$--><!--?lit$747127195$--><path d="M14 10.625H6v-1.25h8v1.25ZM20 10a10 10 0 1 0-10 10 10.011 10.011 0 0 0 10-10Zm-1.25 0A8.75 8.75 0 1 1 10 1.25 8.76 8.76 0 0 1 18.75 10Z"></path><!--?--> </svg></span> <!--?lit$747127195$--> </span> <!--?lit$747127195$--><!--?--><!----><!----><!----> </button>  I want to press it with Playwright in my Python code. I wrote: page.locator('button[aria-label="Toggle Comment Thread"]').first().click()  Sadly, it is giving me the error TypeError: 'Locator' object is not callable. When I don't use first(), it says Error: Error: strict mode violation: locator("button[aria-label=\"Toggle Comment Thread\"]") resolved to 3 elements:  which is right, there are 3 of the buttons on the page. I just want to click the first one.

Answer: Try .first rather than .first(): page.locator('button[aria-label="Toggle Comment Thread"]').first.click()  Runnable example: from playwright.sync_api import sync_playwright # 1.40.0 html = '<button aria-label="Toggle Comment Thread"></button>' with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() page.set_content(html) page.locator('button[aria-label="Toggle Comment Thread"]').first.click() browser.close()  That said, get_by_role is preferred: page.get_by_role("button", name="Toggle Comment Thread").first.click()  And try to avoid .first; find a stricter way to uniquely identify the element (possibly using one of its other many properties) rather than relying on element ordering on the page, which is liable to change unexpectedly. This isn't the only occurrence of this API difference. page.url is also a property, not a function.
Question: .venv/bin/python -m pip uninstall mysqlclient /Users/anentropic/dev/project/.venv/bin/python: No module named pip  and .venv/bin/python Python 3.11.5 (main, Sep 18 2023, 15:04:25) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pip Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'pip'  I thought pip was a builtin for recent Python versions, how is it that I have one where it doesn't exist?

Answer: So... pip isn't actually a "builtin" in recent Python versions What changed is it began to be bundled with Python installs by default, so it's usually available but not necessarily. For whatever reason (perhaps because my venv was created by pdm) mine did not come with it. This can be fixed by: .venv/bin/python -m ensurepip  which installs it After that .venv/bin/python -m pip install etc work ok.
Question: I am trying and failing to read a local .accdb file in my Python 3.11 app, which is running in an python:3.11-alpine container. My Dockerfile executes without errors: FROM python:3.11-alpine EXPOSE 5001 ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 RUN apk update && apk add --no-cache gcc g++ musl-dev unixodbc-dev flex bison gawk COPY requirements.txt . RUN python -m pip install -r requirements.txt RUN apk add --no-cache git autoconf automake libtool gettext-dev make RUN git clone https://github.com/mdbtools/mdbtools.git WORKDIR /mdbtools RUN autoreconf -i -f RUN ./configure --with-unixodbc=/usr --disable-dependency-tracking RUN make RUN make install RUN echo -e "\n[MDBTools]\nDescription=MDBTools Driver\nDriver=/usr/local/lib/odbc/libmdbodbc.so" >> /etc/odbcinst.ini RUN apk add --no-cache nano WORKDIR /app COPY . /app RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app USER appuser CMD ["python", "server.py"]  My Python script (accdb_test.py): import pyodbc import argparse parser = argparse.ArgumentParser(description='Connect to an Access database.') parser.add_argument('db_path', type=str, help='The path to the Access database') args = parser.parse_args() conn_str = ( r'DRIVER={MDBTools};' r'DBQ=' + args.db_path + ';' ) try: conn = pyodbc.connect(conn_str) print("Connection successful!") except pyodbc.Error as e: print("Failed to connect to the database:", e)  I build the container connect to its terminal, than I run the script with this result: /app $ python accdb_test.py /app/input_examples/caesar/MODEL_13-16_R01.ACCDB ['MDBTools'] File not found File not found Unable to locate database Failed to connect to the database: ('HY000', 'The driver did not supply an error!')  The path to the .accdb file is correct, I checked: /app $ ls -l /app/input_examples/caesar/MODEL_13-16_R01.ACCDB -rwxrwxrwx 1 appuser root 47116288 Mar 18 09:29 /app/input_examples/caesar/MODEL_13-16_R01.ACCDB 

Answer: As far as I can tell, mdbtools expects a 2-component connection string, with only a single semicolon. Since you end your connection string with a semicolon, it is looking for a file named /app/input_examples/caesar/MODEL_13-16_R01.ACCDB; and cannot find that file. This can be seen here in the source, it's splitting the connection string into a maximum of 2 components, where any further separators and components get added to the file name. The fix is simple: remove the semicolon: conn_str = ( r'DRIVER={MDBTools};' r'DBQ=' + args.db_path )
Question: How do I get this behaviour: (pl.Series(['abc_remove_def', 'remove_abc_def', 'abc_def_remove']).str.split('_') .map_elements(lambda x: [y for y in x if y != 'remove']).list.join('_') )  Without using the slower map_elements? I have tried using .list.eval and pl.element() but I can't find anything that actually excludes elements from a list by name (i.e. by the word 'remove' in this case)

Answer: list.eval, in combination with filter would work as following: # list_eval (pl .Series(['abc_remove_def', 'remove_abc_def', 'abc_def_remove']).str.split('_') .list.eval(pl.element().filter(pl.element() != 'remove')) )  That said, list.set_difference as suggested by @jqurious is the most straightforward and fastest: # list_set_difference (pl .Series(['abc_remove_def', 'remove_abc_def', 'abc_def_remove']).str.split('_') .list.set_difference(['remove']) )  Output: shape: (3,) Series: '' [list[str]] [ ["abc", "def"] ["abc", "def"] ["abc", "def"] ]  Timings and differences lists of 3 items  lists of 100 items with many duplicates  lists of 100 items without duplicates  NB. the timings exclude the creation of the Series. Additionally, it is important to note that list.set_difference would also remove duplicated values. For instance on: s = pl.Series(['abc_remove_abc_def', 'remove_abc_def']).str.split('_') # output after set_difference shape: (2,) Series: '' [list[str]] [ ["abc", "def"] ["def", "abc"] ] # output for the other approaches shape: (2,) Series: '' [list[str]] [ ["abc", "abc", "def"] ["abc", "def"] ]
Question: Consider the function: def mysum(x)->int: s = 0 for i in x: s += i return s  The argument x can be list[int] or set[int], it can also be d.keys() where d is a dict, it can be range(10), as well as any other iterable, where the item is of type int. What is the correct type-hint for x? My python is 3.10+.

Answer: You can use typing.Iterable: from typing import Iterable def mysum(x: Iterable[int]) -> int: s = 0 for i in x: s += i return s   Edit: typing.Iterable is an alias for collections.abc.Iterable, so you should use that instead, as suggested in the comments.
Question: Suppose, I am reading a csv with hundreds of columns. Now, I know that after a particular column say 'XYZ' all the columns are junk. I want to keep all the columns from the beginning till column 'XYZ' and drop all the columns after column 'XYZ'. In pandas, I may do something like: df.iloc[:, :df.columns.get_loc('XYZ') + 1]  What could be an efficient way in polars?

Answer: The slicing with [:] solution works for eager evaluation (DataFrame), but as far as I know it doesn't really work for LazyFrame. If you want to be able to use lazy evaluation, you can just use DataFrame.select(): # prepare the data df = pl.LazyFrame({ 'ABC': [1,2,3], 'DEF': [4,5,6], 'XYZ': [7,8,9], 'garbage1': [10,11,12], 'garbage2': list('abc') }) df.sink_csv('test.csv')  now we can scan it: df = pl.scan_csv('test.csv') df.select(df.columns[:df.columns.index('XYZ')+1]).collect() ┌─────┬─────┬─────┐ │ ABC ┆ DEF ┆ XYZ │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╡ │ 1 ┆ 4 ┆ 7 │ │ 2 ┆ 5 ┆ 8 │ │ 3 ┆ 6 ┆ 9 │ └─────┴─────┴─────┘  I'm not sure, however, how efficient polars can filter out columns which don't have to be returned, cause they still have to be read from the file.
Question: This is a follow-up on this question. When we use a numpy array with a specific type, it preserves its type following numeric operations. For example adding 1 to a uint32 array will wrap up the value to 0 if needed (when the array contained the max uint32 value) and keep the array of type uint32: import numpy a = numpy.array([4294967295], dtype='uint32') a += 1 # will wrap to 0 print(a) print(a.dtype)  Output: uint32 [0] uint32  This behavior does not hold for an array scalar with the same type: import numpy a = numpy.uint32(4294967295) print(a.dtype) a += 1 # will NOT wrap to 0, and change the scalar type print(a) print(a.dtype)  Output: uint32 4294967296 int64  But according to the array scalars documentation:  The primary advantage of using array scalars is that they preserve the array type ... Therefore, the use of array scalars ensures identical behaviour between arrays and scalars, irrespective of whether the value is inside an array or not.  (emphasys is mine) My question: Why do I observe the above different behavior between arrays and scalars despite the explicit documentation that states they should behave identically ?

Answer: As mentioned in the comments: yes, this documentation is imprecise at best. I think it is referring to the behavior between scalars of the same type: import numpy a = numpy.uint32(4294967295) print(a.dtype) # uint32 a += np.uint32(1) # WILL wrap to 0 with warning print(a) # 0 print(a.dtype) # uint32  The behavior of your example, however, will change due to NEP 50 in NumPy 2.0. So as frustrating as the old behavior is, there's not much to be done but wait, unless you want to file an issue about backporting a documentation change. As documented in the Migration Guide.  The largest backwards compatibility change of this is that it means that the precision of scalars is now preserved consistently... np.float32(3) + 3. now returns a float32 when it previously returned a float64.  I've confirmed that in your example, the type is preserved as expected. import numpy a = numpy.uint32(4294967295) print(a.dtype) # uint32 a += 1 # will wrap to 0 print(a) # 0 print(a.dtype) # uint32 numpy.__version__ # '2.1.0.dev0+git20240318.6059db1'  The second NumPy 2.0 release candidate is out, in case you'd like to try it: https://mail.python.org/archives/list/numpy-discussion@python.org/thread/EGXPH26NYW3YSOFHKPIW2WUH5IK2DC6J/
Question: Given a dataframe with one columm "Name" and two rows. Input: Name ABCD XYZ Output: Name DCBA ZYX  I have searched online and the solution is to reverse the row/column order. Kindly suggest how can I reverse the value in a column in A DataFrame

Answer: You have to use a loop. Assuming strings, you could use apply: df['Name'] = df['Name'].apply(lambda x: x[::-1])  or a list comprehension: df['Name'] = [x[::-1] for x in df['Name']]  Output:  Name 0 DCBA 1 ZYX  If you don't have only strings, a safer approach would be to check for the type: df['Name'] = df['Name'].apply(lambda x: x[::-1] if isinstance(x, str) else x) # or df['Name'] = [x[::-1] if isinstance(x, str) else x for x in df['Name']]
Question: I just learned about enums and thought that they would fit something I'm coding. But when I run this code, I get an error. Am I trying to do something I shouldn't be doing or is this a bug? When trying to groupby a column with enums, I get this error: TypeError: '<' not supported between instances of 'CarBrand' and 'CarBrand' The code: import pandas as pd from enum import Enum class CarBrand(Enum): VOLVO = 'Volvo' BMW = 'BMW' data = { 'brand': [CarBrand.VOLVO, CarBrand.VOLVO, CarBrand.BMW], 'price': [35000, 37000, 45000] } df = pd.DataFrame(data) sum_per_brand = df.groupby('brand').sum('price') print(sum_per_brand)  This is the print I was expecting: brand price BMW 45000 VOLVO 72000

Answer: pd.DataFrame.groupby sorts by default. It works if you use sort=False: sum_per_brand = df.groupby('brand', sort=False).sum('price')  Alternatively, you could use a datatype which supports sorting (like CategoricalDtype).
Question: Suppose I have a function that compares rows in a dataframe: def comp(lhs: pandas.Series, rhs: pandas.Series) -> bool: if lhs.id == rhs.id: return True if abs(lhs.val1 - rhs.val1) < 1e-8: if abs(lhs.val2 - rhs.val2) < 1e-8: return True return False  Now I have a dataframe containing id, val1 and val2 columns and I want to generate group ids such that any two rows for which comp evaluates to true have the group number. How do I do this with pandas? I've been trying to think of a way to get groupby to achieve this but can't think of a way. MRE: example_input = pandas.DataFrame({ 'id' : [0, 1, 2, 2, 3], 'value1' : [1.1, 1.2, 1.3, 1.4, 1.1], 'value2' : [2.1, 2.2, 2.3, 2.4, 2.1] }) example_output = example_input.copy() example_output.index = [0, 1, 2, 2, 0] example_output.index.name = 'groups' 

Answer: You want to cluster rows that either belong to the same group, or that are close in distance. For that compute the distance with scipy.spatial.distance.pdist to identify the close points, and create a graph with networkx to identify the connected components: import networkx as nx import pandas as pd from scipy.spatial.distance import pdist from itertools import combinations # example input df = pandas.DataFrame({ 'id' : [0, 1, 2, 2, 3], 'value1' : [1.1, 1.2, 1.3, 1.4, 1.1], 'value2' : [2.1, 2.2, 2.3, 2.4, 2.1] }) thresh = 1e-8 cols = ['value1', 'value2'] # create graph based on already connected ids G = nx.compose_all(map(nx.path_graph, df.index.groupby(df['id']).values())) # add pairs of values with distance below threshold as edges G.add_edges_from(pd.Series(combinations(df.index, 2)) [pdist(df[cols])<thresh] ) # form groups based on the connected components groups = {n: i for i, c in enumerate(nx.connected_components(G)) for n in c} # {0: 0, 4: 0, 1: 1, 2: 2, 3: 2} # update index based on above dictionary df.index = df.index.map(groups)  Output:  id value1 value2 0 0 1.1 2.1 1 1 1.2 2.2 2 2 1.3 2.3 2 2 1.4 2.4 0 3 1.1 2.1  Graph (only based on IDs ; numbers are the original indices):  Graph (after taking the distance into account):
Question: Here is what I am trying to do. I have the following DataFrame in pandas: import numpy as np import pandas as pd n_cols = 3 n_samples = 4 df = pd.DataFrame(np.arange(n_samples * n_cols).reshape(n_samples, n_cols), columns=list('ABC')) print(df)  output:  A B C 0 0 1 2 1 3 4 5 2 6 7 8 3 9 10 11  I have a category to which each sample (row) belongs: cat = pd.Series([1,1,2,2])  And I have a reference row related to each category: df_ref = pd.DataFrame(np.zeros((2, n_cols)), index=[1,2], columns=list('ABC')) df_ref.loc[1] = 10 print(df_ref)  output:  A B C 1 10.0 10.0 10.0 2 0.0 0.0 0.0  How do I do the following in a more elegant way (e.g., using groupby and transform): result = df.copy() for i in range(n_cols): result.iloc[i] = df.iloc[i] - df_ref.loc[cat[i]] print(results)  output:  A B C 0 -10 -9 -8 1 -7 -6 -5 2 6 7 8 3 9 10 11  I thought something like this should work: df.groupby(cat).transform(lambda x: x - df_ref.loc[x.GROUP_NAME])  where x.GROUP_NAME is accessing the name of the group on which transform is operating. In the pandas documentation about transform it is written: "Each group is endowed the attribute ‘name’ in case you need to know which group you are working on." I tried to access x.name, but that gives the name of a column, not the name of the group. So I don't understand what this documentation is referring to.

Answer: No need for grouby, just reindex df_ref and convert to array: df -= df_ref.reindex(cat).values  Or, for a copy: out = df.sub(df_ref.reindex(cat).values)  Note that your approach would work with groupby.apply: out = df.groupby(cat, group_keys=False).apply(lambda x: x - df_ref.loc[x.name])  Output:  A B C 0 -10.0 -9.0 -8.0 1 -7.0 -6.0 -5.0 2 6.0 7.0 8.0 3 9.0 10.0 11.0
Question: I'm using time_marker = {{execution_date.in_timezone('Europe/Amsterdam')}} in my dag.py program. I'm trying to keep the date part in execution date, and set the time to "T00:00:00" So whenever during the execution date it runs, the time_marker will always be for example 20240115T00:00:00 How should I do this? I tried to use pendulum.parse but didn't work out how to do this. Thanks.

Answer: Add a custom macro to the DAG definition Function: def format_execution_date(execution_date): amsterdam_time = execution_date.in_timezone('Europe/Amsterdam') midnight_amsterdam_time = amsterdam_time.start_of('day') return midnight_amsterdam_time.format('YYYYMMDDT00:00:00')  Use function: with DAG( ... user_defined_macros={'format_execution_date': format_execution_date}, ) as dag: ...  In your task, you can used by: params={ 'time_marker': '{{ format_execution_date(execution_date) }}' }