eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am following an introduction tutorial for plotly and I am facing a problem, using Jupyter. This is the code below: import numpy as np import pandas as pd import cufflinks as cf import chart_studio.plotly as py import plotly.tools as tls import plotly.graph_objs as go tls.set_credentials_file(username = "xxx", api_key = "yyyy")  And the error I am getting. -------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[21], line 1 ----> 1 tls.set_credentials_file(username = "xxxx", api_key = "yyy") AttributeError: module 'plotly.tools' has no attribute 'set_credentials_file'  All the other questions I am seeing on SO are from some years ago and I am afraid the documentation of the package is updated. Can someone help?

Answer: As it was mentioned by @Sotos in the comment, the set_credentials_file file is no longer available in plotly.tools. You should use the chart_studio instead: https://pypi.org/project/chart-studio/ and it should work for you: import chart_studio.tools as tls tls.set_credentials_file(username='xxx', api_key='yyyy')  Also have a look at the current Getting Started doc: https://plotly.com/python/getting-started-with-chart-studio/.
Question: This works: f = lambda t: {'x': 1, 'y': "abc"} df = pl.DataFrame( {'c': [f(0)] } )  shape: (1, 1) ┌───────────┐ │ c │ │ --- │ │ struct[2] │ ╞═══════════╡ │ {1,"abc"} │ └───────────┘  >>> df.schema # Schema([('c', Struct({'x': Int64, 'y': String}))])  While using map_rows with same f and schema dies with: RuntimeError: BindingsError: "Could not determine output type" (df.with_columns('c') .map_rows(f, return_dtype= pl.Struct([pl.Field('x', pl.Int64), pl.Field('y', pl.String)]) ) ) 

Answer: There are really two different apply methods that are relevant here. There's DataFrame.apply and there's Expr.apply. When you do: df.with_columns('c')  The output of that is a DataFrame that includes c. The with_columns isn't doing anything except making sure c exists and throwing an error if it doesn't. Therefore when you chain an apply to that you're invoking the DataFrame.apply method. That method is equivalent to doing a for loop on df.rows where the input to the function is a tuple of the values of each column one row at a time. It's expecting the output of the function to be a tuple so if your f was: f1 = lambda t: ({'x': 1, 'y': "abc"},)  then doing df.apply(f1) would work. On the other hand if you invoke the Expr.apply by doing df.with_columns(pl.col('c').apply(f))  then that works too because the Expr.apply is equivalent to running a for loop on all the values of the column from which it was invoked. Obligatory anti-apply rant: If you're using apply in polars then you're not getting any benefit from the library. At that point, you're just doing a python for-loop with all of the inherent slowness.
Question: I have a dataframe with columns A and B. Column A has non continuous data where some of the rows are NAN and B has continuous data. I would like to create a third column where for each set of A rows with NAN it will have the sum of values in those same rows in B + the next valid value in B. All other values in C should be NAN for NAN in A AND the value of B for rows following a valid number in A. Example: data = { 'A': [1, 1, None, None, 2, 5, None, None,3 ,4, 3, None , 5], 'B': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130]}  Everything works fine except for the rows where I need the sum of B + next valid value in B. I use the following code. I have this code but is seems it's a mess by now. `result = df.groupby(df['A'].isnull().cumsum())['B'].sum().reset_index() df_result = pd.DataFrame({'C': result['Pumped']}) df_result.loc[1:, 'C'] -= result.loc[0, 'Pumped'] df.loc[~mask, 'C'] = df.loc[~mask, 'Pumped'] valid_rows_after_nan = df['dWL'].notnull() & mask.shift(1).fillna(False) df.loc[valid_rows_after_nan, 'C'] = df_result print(df)`  I would like the output to look like this: `data = { 'A': [1, 1, None, None, 2, 5, None, None,3 ,4, 3, None , 5], 'B': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130], 'C': [10, 20, None, None, 120, 60, None, None, 240, 100, 110, None, 5] } 

Answer: A simple version using groupby.transform: # identify the non-NA and reverse m = df.loc[::-1, 'A'].notna() # group the preceding NA, sum, mask where NA df['C'] = df.groupby(m.cumsum())['B'].transform('sum').where(m)  Output:  A B C 0 1.0 10 10.0 1 1.0 20 20.0 2 NaN 30 NaN 3 NaN 40 NaN 4 2.0 50 120.0 5 5.0 60 60.0 6 NaN 70 NaN 7 NaN 80 NaN 8 3.0 90 240.0 9 4.0 100 100.0 10 3.0 110 110.0 11 NaN 120 NaN 12 5.0 130 250.0
Question: When I run my program, I immediately get an AttributeError when it's setting up the argument parser: Traceback (most recent call last): File "blah.py", line 55, in <module> parser.add_argument('--select-equal', '--equal', nargs=2, metavar=[ 'COLUMN', 'VALUE' ], action='append', dest='filter_equality_keys_and_values', default=[], help='Only return rows for which this column is exactly equal to this value.') File "…/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/argparse.py", line 1346, in add_argument chars = self.prefix_chars AttributeError: 'str' object has no attribute 'prefix_chars'  Well, that sounds reasonable. strs indeed don't have a prefix_chars attribute. Clearly the exception message is correct. Except I can't parse arguments, or even get to parsing arguments, because of this exception. Why is this throwing an exception? Why is it trying to get prefix_chars of a str in the first place? The exception is coming from inside the Python module; is this a bug in Python or something I'm doing wrong?

Answer: It is, in fact, something I was doing wrong. The clue is that the method is trying to access an attribute of an object that doesn't have such an attribute—and it's trying to do so on self. The first argument to my call to add_argument is indeed a string ('--select-equal'). The only way that string could end up being self is if I were calling add_argument on the ArgumentParser class and not an ArgumentParser object. And indeed, scrolling up a couple lines in my own code, there's the problem:  import argparse parser = argparse.ArgumentParser  I need to instantiate ArgumentParser, which means calling the class (ArgumentParser()), not just stash the class in a variable. For want of a (), the exception was thrown.
Question: I create a Docker image based on rocker/shinyversewith the Dockerfile: # File: Dockerfile FROM rocker/shiny-verse:4.2.2 RUN echo "apt-get start" RUN apt-get update && apt-get install -y \ python3 \ python3-pip # install R packages RUN R -e "install.packages('remotes')" RUN R -e "install.packages('reticulate')" RUN R -e "install.packages('tidyverse')" # Install Python packages datetime and zeep RUN python3 -m pip install datetime zeep # Set the environment variable to use Python3 ENV RETICULATE_PYTHON /usr/bin/python3  Then I have the simple R file: library("tidyverse") library(reticulate) # Call a simple Python command to calculate 3*5 py_run_string("z = 3+4") py$z %>% print()  After I launched the container, I want to run this R script with the shell command: docker exec shiny_new Rscript /home/shiny/ETL/reticulate_test.R  but I get the following error: Error: Python shared library not found, Python bindings not loaded. Use reticulate::install_miniconda() if you'd like to install a Miniconda Python environment. Execution halted  It fails when executing the python I am unsure how to setup the Python in such a way that I can use python code via reticulate in my R script. Does anybody have an idea where I go wrong in setting up the Docker image?

Answer: Folowing this issue, it seems pyenv installs Python without the Python shared library. You should try to add:  reticulate::install_python() which wraps pyenv and set --enable-shared option  or  set the --enable-shared variable as suggested here
Question: I am trying to get a simple vector store (chromadb) to embed texts using the add_texts method with langchain, however I get the following error despite successfully using the OpenAI package with a different simple langchain scenario: ValueError: You must provide embeddings or a function to compute them  Code: from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import Chroma db = Chroma() texts = [ """ One of the most common ways to store and search over unstructured data is to embed it and store the resulting embedding vectors, and then at query time to embed the unstructured query and retrieve the embedding vectors that are 'most similar' to the embedded query. A vector store takes care of storing embedded data and performing vector search for you. """, """ Today's applications are required to be highly responsive and always online. To achieve low latency and high availability, instances of these applications need to be deployed in datacenters that are close to their users. Applications need to respond in real time to large changes in usage at peak hours, store ever increasing volumes of data, and make this data available to users in milliseconds. """, ] db.add_texts(texts, embedding_function=OpenAIEmbeddings()) 

Answer: embedding_function need to be passed when you construct the object of Chroma. source : Chroma class Class Code so your code would be: from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import Chroma db = Chroma(embedding_function=OpenAIEmbeddings()) texts = [ """ One of the most common ways to store and search over unstructured data is to embed it and store the resulting embedding vectors, and then at query time to embed the unstructured query and retrieve the embedding vectors that are 'most similar' to the embedded query. A vector store takes care of storing embedded data and performing vector search for you. """, """ Today's applications are required to be highly responsive and always online. To achieve low latency and high availability, instances of these applications need to be deployed in datacenters that are close to their users. Applications need to respond in real time to large changes in usage at peak hours, store ever increasing volumes of data, and make this data available to users in milliseconds. """, ] db.add_texts(texts)  and result you will see as ['58f12150-2bc4-11ee-9ff5-ac87a32b530e', '58f12240-2bc4-11ee-9ff5-ac87a32b530e']
Question: I use selenium in my python project. Few days ago program worked correctly but today I ran my script and it caused this warning. Script: from selenium import webdriver import chromedriver_autoinstaller chromedriver_autoinstaller.install() options = webdriver.ChromeOptions() prefs = {"profile.default_content_setting_values.notifications": 1} options.add_argument('--ignore-certificate-errors') options.add_argument('--ignore-ssl-errors') options.add_experimental_option("prefs", prefs) options.add_argument("--disable-blink-features=AutomationControlled") options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option("useAutomationExtension", False) driver = webdriver.Chrome(options=options) driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})") driver.execute_cdp_cmd("Network.setUserAgentOverride", {"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"})  I tried reinstall chromedriver but it didn't help

Answer: Selenium Manager is now fully included with selenium 4.10.0, so this is all you need: from selenium import webdriver from selenium.webdriver.chrome.service import Service service = Service() options = webdriver.ChromeOptions() driver = webdriver.Chrome(service=service, options=options) # ... driver.quit()  If the driver isn't found on your system PATH, Selenium Manager will automatically download it.  If you're wondering why you're now seeing this error, it's because https://chromedriver.chromium.org/downloads only goes up to version 114 due to driver restructuring by the Chromium Team for the new Chrome-for-Testing.
Question: Given the following array: array = [-1, -1, -1, -1, -1, -1, 3, 3, -1, 3, -1, -1, 2, 2, -1, -1, 1, -1] indexes 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17  I need to find the indexes where the same number appears. In this example this would return a list of lists like this: list(list(), list(16), list(12, 13), list(6, 7, 9), list() etc...) 0 1 \ 2 3 4 ^ \ \ \ the index in the array at which "1" appears \ \ the numbers in the array  how would one do this in numpy? the number 1 appears at index 16 the number 2 appears at indexes 12, 13 etc. NOTES based on comments:  -1 can be ignored, i'm only interested in the rest  array has ~50 elements with values up to int(500)  this function will be called 6000+ times.  

Answer: For a solution in O(n) time*, use a dictionary to collect the indices: # collect positions per value for each item d = {} for i, x in enumerate(array): d.setdefault(x, []).append(i) # sort the output (optional) out = {k: d[k] for k in sorted(d)}  Output: {-1: [0, 1, 2, 3, 4, 5, 8, 10, 11, 14, 15, 17], 1: [16], 2: [12, 13], 3: [6, 7, 9]}  * + O(k*log(k)) where k is the number of unique values if you need a sorted output For a list of lists: out = [d.get(k, []) for k in range(min(d), max(d)+1)] # or for only positive values out = [d.get(k, []) for k in range(1, max(d)+1)]  Output: [[0, 1, 2, 3, 4, 5, 8, 10, 11, 14, 15, 17], [], [16], [12, 13], [6, 7, 9]]  alternative Or a very simple approach if you pre initialize the output: out = [[] for i in range(max(array)+1)] for i, x in enumerate(array): out[x].append(i)  comparison of all approaches pure python is the fastest here. The initial array was generated using np.random.randint(0, k, size=n).tolist() where n is the length of the array and k the maximum value in the array. With k=4:  With k=100: We can now see the quadratic behavior of @TalhaTayyab/@Stitt/@PaulS approaches.  With k=10_000: We can note that numpy is a bit faster for relatively small arrays (when the values are likely unique).
Question: from a data frame df1 =     Area Sequence X Y     A 2 604582.25 320710   A 1 604590.25 320704.75   A 3 604579.25 320710   B 2 536584.47 176977.83   B 1 536570 176996.43   C 1 509202.13 307995.99   C 2 509205.3 307951.24     Need to generate into df1 =     Area XY_by_sequence     A 604590.25 320704.75 , 604582.25 320710 , 604579.25 320710   B 536570 176996.43 , 536584.47 176977.83   C 509202.13 307995.99 , 509205.3 307951.24    

Answer: Try: x = ( df.set_index(["Area", "Sequence"]) .stack() .groupby(level=[0, 1]) .agg(lambda x: " ".join(map(str, x))) .groupby(level=0) .agg(", ".join) .reset_index(name="XY_by_sequence") ) print(x)  Prints:  Area XY_by_sequence 0 A 604590.25 320704.75, 604582.25 320710.0, 604579.25 320710.0 1 B 536570.0 176996.43, 536584.47 176977.83 2 C 509202.13 307995.99, 509205.3 307951.24
Question: How do I do a type hint for attribute that might not exist on the object in Python? For example: class X: __slots__ = ('attr',) attr: int # either int or doesn't exist - what do I put here?  So, concretely: >>> x = X() >>> x.attr Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'X' object has no attribute 'attr' >>> x.attr = 1 >>> x.attr 1  ...but there are also cases where __slots__ isn't involved.

Answer: The answer currently appears to be "You can't". I've started a discussion thread for this in the Python Typing repo, which may get more visibility.
Question: I have the following model in pydantic (Version 2.0.3) from typing import Tuple from pydantic import BaseModel class Model(BaseModel): test_field: Tuple[int]  But when I enter model = Model(test_field=(1,2))  I get as error: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/code.py", line 90, in runcode exec(code, self.locals) File "<input>", line 1, in <module> File "/Users/tobi/Documents/scraiber/z_legacy/fastapi_test_app/venv/lib/python3.10/site-packages/pydantic/main.py", line 150, in __init__ __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__) pydantic_core._pydantic_core.ValidationError: 1 validation error for Model test_field Tuple should have at most 1 item after validation, not 2 [type=too_long, input_value=(1, 2), input_type=tuple] For further information visit https://errors.pydantic.dev/2.0.3/v/too_long  Do you know how I can fix that?

Answer: Following @Tim Robert's Answer, the linked PR suggests using the Ellipsis ... is the syntax you're after! https://github.com/pydantic/pydantic/pull/512/files class Model(BaseModel): test_field: Tuple[int, ...]  >>> Model(test_field=(1,2)) Model(test_field=(1, 2))   Additionally, and though I don't think it's really advisable (prefer codegen or reconsider the design), you can generate and expand given an exact count of the fields count = 5 class Model_Five(BaseModel): test_field: Tuple[*([int]*count)]  >>> Model_Five(test_field=(1,2,3,4,5)) Model_Five(test_field=(1, 2, 3, 4, 5)) >>> Model_Five(test_field=(1,2,3,4)) [..] omitted test_field.4 Field required [type=missing, input_value=(1, 2, 3, 4), input_type=tuple] For further information visit https://docs.pydantic.dev/dev/errors/validation_errors/#missing  Finally, there might be cases where you need a longer form like *(cls for _ in range(count)) (or the even more troublesome for sustainability *(some_class_factory() for _ in range(count))) to expressly avoid having the inner values refer to the same object
Question: I have a problem I'm trying to solve but can't figure it out. I have something similar to this table: df = pl.from_repr(""" ┌─────┬────────────┬───────┐ │ id ┆ date ┆ sales │ │ --- ┆ --- ┆ --- │ │ i64 ┆ str ┆ f64 │ ╞═════╪════════════╪═══════╡ │ 1 ┆ 2023-01-01 ┆ 10.0 │ │ 1 ┆ 2023-02-01 ┆ 20.0 │ │ 1 ┆ 2023-03-01 ┆ 30.0 │ │ 1 ┆ 2023-05-01 ┆ 40.0 │ │ 2 ┆ 2023-02-01 ┆ 50.0 │ │ 2 ┆ 2023-03-01 ┆ 60.6 │ │ 2 ┆ 2023-04-01 ┆ 70.2 │ │ 3 ┆ 2023-01-01 ┆ 80.5 │ │ 3 ┆ 2023-02-01 ┆ 90.0 │ └─────┴────────────┴───────┘ """)  as you can see, for each id i have dates and sales. I want to get for every id all dates from the minimum date in the data frame up to the maximum(included). in addition, i want to fill in the sales column with 0 and the id column with the matching id, so it looks like this: ┌─────┬────────────┬───────┐ │ id ┆ date ┆ sales │ │ --- ┆ --- ┆ --- │ │ i32 ┆ date ┆ f64 │ ╞═════╪════════════╪═══════╡ │ 1 ┆ 2023-01-01 ┆ 10.0 │ │ 1 ┆ 2023-01-02 ┆ 20.0 │ │ 1 ┆ 2023-01-03 ┆ 30.0 │ │ 1 ┆ 2023-01-04 ┆ 0.0 │ │ 1 ┆ 2023-01-05 ┆ 40.0 │ │ 2 ┆ 2023-01-01 ┆ 0.0 │ │ 2 ┆ 2023-01-02 ┆ 50.0 │ │ 2 ┆ 2023-01-03 ┆ 60.6 │ │ 2 ┆ 2023-01-03 ┆ 70.2 │ │ 2 ┆ 2023-01-04 ┆ 0.0 │ │ 2 ┆ 2023-01-05 ┆ 0.0 │ │ 3 ┆ 2023-01-01 ┆ 80.5 │ │ 3 ┆ 2023-01-02 ┆ 90.0 │ │ 3 ┆ 2023-01-03 ┆ 0.0 │ │ 3 ┆ 2023-01-04 ┆ 0.0 │ │ 3 ┆ 2023-01-05 ┆ 0.0 │ └─────┴────────────┴───────┘  so on and so forth. i've tried to create a new dataframe by using the pl.date_range function and then to join it against the main data, by using outer or cross, but to no avail, since it doesn't compute against each id. maybe you have any ideas on how to go about it? many thanks in advance for any input!

Answer: Try: def fn(x, r): print(pl.DataFrame({"id": x["id"][0], "date": r})) print(x) return ( pl.DataFrame({"id": x["id"][0], "date": r}) .join(x, on="date", how="left")[["id", "date", "sales"]] .with_columns(pl.col("sales").fill_null(strategy="zero")) ) # convert to date df = df.with_columns(pl.col("date").str.to_date("%Y-%d-%m")) # get min, max date mn, mx = df["date"].min(), df["date"].max() # construct the range r = pl.date_range(mn, mx, "1d", eager=True) # group by "id" and fill the missing dates df = df.group_by("id", maintain_order=True).map_groups(lambda x: fn(x, r)) with pl.Config(tbl_rows=-1): print(df)  Prints: shape: (15, 3) ┌─────┬─────────────────────┬───────┐ │ id ┆ date ┆ sales │ │ --- ┆ --- ┆ --- │ │ i64 ┆ datetime[μs] ┆ f64 │ ╞═════╪═════════════════════╪═══════╡ │ 1 ┆ 2023-01-01 00:00:00 ┆ 10.0 │ │ 1 ┆ 2023-01-02 00:00:00 ┆ 20.0 │ │ 1 ┆ 2023-01-03 00:00:00 ┆ 30.0 │ │ 1 ┆ 2023-01-04 00:00:00 ┆ 0.0 │ │ 1 ┆ 2023-01-05 00:00:00 ┆ 40.0 │ │ 2 ┆ 2023-01-01 00:00:00 ┆ 0.0 │ │ 2 ┆ 2023-01-02 00:00:00 ┆ 50.0 │ │ 2 ┆ 2023-01-03 00:00:00 ┆ 60.6 │ │ 2 ┆ 2023-01-04 00:00:00 ┆ 70.2 │ │ 2 ┆ 2023-01-05 00:00:00 ┆ 0.0 │ │ 3 ┆ 2023-01-01 00:00:00 ┆ 80.5 │ │ 3 ┆ 2023-01-02 00:00:00 ┆ 90.0 │ │ 3 ┆ 2023-01-03 00:00:00 ┆ 0.0 │ │ 3 ┆ 2023-01-04 00:00:00 ┆ 0.0 │ │ 3 ┆ 2023-01-05 00:00:00 ┆ 0.0 │ └─────┴─────────────────────┴───────┘
Question: Suppose I have an expression like "col3 = col2 + col1" so pandas we can directly call pandas.dataframe.eval() but in polars I cannot find such method. I have series.eval in polars but no luck as I want evaluate user given expression on a dataframe.

Answer: Acception strings You can pass SQL to pl.sql_expr. df = pl.DataFrame({ "col1": [1, 2], "col2": [1, 2], }) df.select( pl.sql_expr("col2 + col1 as col3") )  Or you can run a complete SQL query with pl.sql pl.sql("SELECT col2 + col1 as col3 FROM df").collect()  shape: (2, 1) ┌──────┐ │ col3 │ │ --- │ │ i64 │ ╞══════╡ │ 2 │ │ 4 │ └──────┘  Accept expressions directly  i want evaluate user given expression on a dataframe.  I would accept pl.Expression directly instead of strings. This gives more type safety than strings and probably also a better user experience as you can have autocomplete and the IDE may show available methods/arguments.
Question: I tried to load Llama-2-7b-hf LLM with QLora with the following code: model_id = "meta-llama/Llama-2-7b-hf" tokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=True) # I have permissions. model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, quantization_config=bnb_config, device_map="auto", use_auth_token=True) model.gradient_checkpointing_enable() model = prepare_model_for_kbit_training(model) config = LoraConfig( r=8, lora_alpha=32, target_modules=[ "query_key_value", "dense", "dense_h_to_4h", "dense_4h_to_h", ], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM" ) model = get_peft_model(model, config) # got the error here  I got this error:  File "/home/<my_username>/.local/lib/python3.10/site-packages/peft/tuners/lora.py", line 333, in _find_and_replace raise ValueError( ValueError: Target modules ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h'] not found in the base model. Please check the target modules and try again.  How can I solve this? Thank you!

Answer: the strings in target_modules are different from models to models, you can debug the model = AutoModelForCausalLM.from_pretrained(model_id)  and look into the model then you can find what are the linear layers names and input them into the target_modules
Question: I dont know if I have missed it, but this is not clear to me. I already have miniconda on my machine and I want now to install mamba. How this should be done please, I am supposed to download/run the correct mambaforge installer? Does this co-exist happily side by side with conda or you are supposed to have just one of them on your system. Doing something like conda install mamba is not recommended

Answer: Mamba is a replacement conda package manager. They do not co-exist happily as they both rely on a base environment for their dependencies. The recommended way is to install mambaforge or micromamba separately as a replacement. documentation for Mamba: https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html documentation for Micromamba: https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html