eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to install the pytoch version 1.0.0, but i couldn't able to do that I tried pip install torch===1.0.0 -f https://download.pytorch.org/whl/torch_stable.html  its not wotking I'm getting a error Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Looking in links: https://download.pytorch.org/whl/torch_stable.html ERROR: Could not find a version that satisfies the requirement torch===1.0.0 (from versions: 1.7.1, 1.7.1+cpu, 1.7.1+cu101, 1.7.1+cu110, 1.7.1+cu92, 1.7.1+rocm3.7, 1.7.1+rocm3.8, 1.8.0, 1.8.0+cpu, 1.8.0+cu101, 1.8.0+cu111, 1.8.0+rocm3.10, 1.8.0+rocm4.0.1, 1.8.1, 1.8.1+cpu, 1.8.1+cu101, 1.8.1+cu102, 1.8.1+cu111, 1.8.1+rocm3.10, 1.8.1+rocm4.0.1, 1.9.0, 1.9.0+cpu, 1.9.0+cu102, 1.9.0+cu111, 1.9.0+rocm4.0.1, 1.9.0+rocm4.1, 1.9.0+rocm4.2, 1.9.1, 1.9.1+cpu, 1.9.1+cu102, 1.9.1+cu111, 1.9.1+rocm4.0.1, 1.9.1+rocm4.1, 1.9.1+rocm4.2, 1.10.0, 1.10.0+cpu, 1.10.0+cu102, 1.10.0+cu111, 1.10.0+cu113, 1.10.0+rocm4.0.1, 1.10.0+rocm4.1, 1.10.0+rocm4.2, 1.10.1, 1.10.1+cpu, 1.10.1+cu102, 1.10.1+cu111, 1.10.1+cu113, 1.10.1+rocm4.0.1, 1.10.1+rocm4.1, 1.10.1+rocm4.2, 1.10.2, 1.10.2+cpu, 1.10.2+cu102, 1.10.2+cu111, 1.10.2+cu113, 1.10.2+rocm4.0.1, 1.10.2+rocm4.1, 1.10.2+rocm4.2, 1.11.0, 1.11.0+cpu, 1.11.0+cu102, 1.11.0+cu113, 1.11.0+cu115, 1.11.0+rocm4.3.1, 1.11.0+rocm4.5.2, 1.12.0, 1.12.0+cpu, 1.12.0+cu102, 1.12.0+cu113, 1.12.0+cu116, 1.12.0+rocm5.0, 1.12.0+rocm5.1.1, 1.12.1, 1.12.1+cpu, 1.12.1+cu102, 1.12.1+cu113, 1.12.1+cu116, 1.12.1+rocm5.0, 1.12.1+rocm5.1.1, 1.13.0, 1.13.0+cpu, 1.13.0+cu116, 1.13.0+cu117, 1.13.0+cu117.with.pypi.cudnn, 1.13.0+rocm5.1.1, 1.13.0+rocm5.2, 1.13.1, 1.13.1+cpu, 1.13.1+cu116, 1.13.1+cu117, 1.13.1+cu117.with.pypi.cudnn, 1.13.1+rocm5.1.1, 1.13.1+rocm5.2, 2.0.0, 2.0.0+cpu, 2.0.0+cpu.cxx11.abi, 2.0.0+cu117, 2.0.0+cu117.with.pypi.cudnn, 2.0.0+cu118, 2.0.0+rocm5.3, 2.0.0+rocm5.4.2) ERROR: No matching distribution found for torch===1.0.0 

Answer: It seems like your Python3 version is newer (>3.7) than what the version of torch you're trying to install supports. Available versions of torch start from 1.7.1. Consider using an older version of Python3 (<=3.7). To do this, I recommend installing conda or miniconda from here. Then, run: conda create -n myenv python=3.6.9 Load your env: conda activate myenv Try install torch again: pip install torch===1.0.0 -f https://download.pytorch.org/whl/torch_stable.html Regards.
Question: After running the following code: import torch from accelerate import infer_auto_device_map, init_empty_weights from transformers import AutoConfig, AutoModelForCausalLM config = AutoConfig.from_pretrained("facebook/opt-13b") with init_empty_weights(): model = AutoModelForCausalLM.from_config(config) device_map = infer_auto_device_map(model, no_split_module_classes=["OPTDecoderLayer"], dtype="float16") model = AutoModelForCausalLM.from_pretrained( "facebook/opt-13b", device_map=device_map, offload_folder="offload", offload_state_dict=True, torch_dtype=torch.float16)  I end up with the following error: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /tmp/ipykernel_23/3733710460.py in <module> 4 offload_folder="offload", 5 offload_state_dict=True, ----> 6 torch_dtype=torch.float16) /opt/conda/lib/python3.7/site-packages/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 463 model_class = _get_model_class(config, cls._model_mapping) 464 return model_class.from_pretrained( --> 465 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs 466 ) 467 raise ValueError( /opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs) 2527 # Dispatch model with hooks on all devices if necessary 2528 if device_map is not None: -> 2529 dispatch_model(model, device_map=device_map, offload_dir=offload_folder, offload_index=offload_index) 2530 2531 if output_loading_info: TypeError: dispatch_model() got an unexpected keyword argument 'offload_index'  I'm trying to load a large model from hugging face, and to do that, I split the weights into the GPUs from Kaggle. I don't understand exactly what is going on here, and how to deal with this issue.

Answer: please try updating your accelerate package, for example pip install accelerate==0.18.0
Question: I'm trying to scrape a Bing dict page https://cn.bing.com/dict/search?q=avengers Here is the code import requests from bs4 import BeautifulSoup url = "https://cn.bing.com/dict/search?q=avengers" headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3" } response = requests.get(url, headers=headers) soup = BeautifulSoup(response.content, "html.parser") examples = soup.find_all("div", class_="sen_en b_regtxt") for example in examples: print(example.text.strip())  In particular, I'm trying scrape all the example sentences on that page, which is contained in a div with class sen_en b_regtxt However, response.content doesn't even contain one example sentence in it, what am I missing? PS, access to the page doesn't need login  With @Artur Chukhrai's help, I also tried with selenium and I got "No results found for avenger"  However, if I access the url "cn.bing.com/dict" first, and then put the keyword in the search box, I would get the result page.

Answer: A small modification to Arthur Chukhrai's answer works, by loading https://cn.bing.com/dict and then writing the text in the search box: from selenium import webdriver from selenium.webdriver.common.by import By from bs4 import BeautifulSoup import time url = "https://cn.bing.com/dict/" # Start a new Selenium web driver instance driver = webdriver.Chrome() driver.get(url) # Wait for the page to load time.sleep(5) # Write text in search box search_box = driver.find_element(By.CLASS_NAME, value="b_searchbox") search_box.send_keys("avengers\n") # Wait for the page to load time.sleep(5) # Get the page source after it has fully loaded html = driver.page_source soup = BeautifulSoup(html, "html.parser") # Find and print the examples of the word examples = soup.select(".sen_en") for example in examples: print(example.text.strip()) # Quit the web driver instance driver.quit()
Question: I'm using the python module ratelimit to throttle a function, which calls a rest api, I need to apply throttle based on the method of the requests, e.g. for PUT/POST/DELETE 1 per 10s, for GET 5 per 1s, how can I achieve this without breaking the function into two? from ratelimit import limits, sleep_and_retry @sleep_and_retry @limits(calls=1 if method != 'GET' else 5, period=10 if method != 'GET' else 1) def callrest(method, url, data): ...  Is it possible to do this?

Answer: You don't have to re-invent the wheel by creating your own rate limiter when ratelimit already works well. To apply different rate limits based on the method argument passed in, make a decorator that creates two ratelimit.limits-decorated functions from the given function--one decorated with arguments needed by the GET method, and the other decorated with those needed by non-GET methods. Then make the decorator return a wrapper function that calls one of the two decorated functions above according to the value of the method argument: from ratelimit import limits, sleep_and_retry def limits_by_method(func): def wrapper(method, *args, **kwargs): return (get if method == 'GET' else non_get)(method, *args, **kwargs) get = limits(calls=5, period=1)(func) non_get = limits(calls=1, period=10)(func) return wrapper @sleep_and_retry @limits_by_method def callrest(method, url, data): ...  Demo: https://replit.com/@blhsing/ExoticWretchedAdvance
Question: The question is simple, but I have googled a lot of methods, and there no such solution as: import svg-render-library figure = svg-render-library.open('test.svg') figure.render()  Is there any simple methods to display an SVG image using only python libraries? I am asking about rendering the SVG image without any conversion to another formats and to render using pure python, without any 3rd party software. As I have tried, this seems impossible for now. As built-in python I mean - only python packages available through pip, so it is not necessary to install/compile anything else. And to render I mean to show inside window which part of the python, not the browser or any external software. At least I have currently working solution my question on Stack Overflow.

Answer: Currently, there is no method to render natively cross-platform with just the standard library (ie. some python distributions for OSX do not include tkinter by default). Ergo, there is no good way to do this. AFAIK, there are no other ways to do this maintaining your described API without writing your own code or reaching out to non-standard library modules. If you still are 100% set on doing it with pure python and the standard library, you have tkinter, and don't care about writing your own implementation, then proceed. If you are talking about rendering in the context of displaying an SVG in a window, then your best bet would be to utilize the tkinter and xml modules. SVG is just an XML file, so xml.minidom should be able to parse an svg file. You can then extract the height and width from the dom, draw each element onto a tkinter.Canvas widget, and then have tkinter render the window. You will need to either pre-calculate transparencies or handle that while managing the layering. Another option is to use the turtle package which wraps around tkinter. If the SVG is just a path, then this should be able to draw that path fairly straight forward. If you are willing to reach out beyond the standard library, then cairosvg or svglib both can easily handle this task. cairosvg is a bit of a bear if you aren't used to installing system libraries, but svglib is a pure python implementation.
Question: The table's last column (action) contains pc.button, that can get data in the row that the button residing in, and then send that data to a State variable. Some thing like this:     name age job action     John 20 Developer Update   May 23 Designer Update     I have tried pc.list like this: return pc.center( pc.list([ ["name","age","job","action"], ["John",20, "Developer", pc.button("Update", on_click=State.update_employee)], ["May",23, "Designer", pc.button("Update", on_click=State.update_employee)], ]) )  but pc.list can only take in python primitive types (str,int,float...). Is there any way to add multiple components progamatically to another Pynecone component

Answer: https://youtube.com/shorts/u-TUSQ9DkCw <-- Example here I write the following full example for what you want. The simple answer is to use table_container. But we need to care about some detail. from pcconfig import config import pynecone as pc class Member(pc.Model, table=True): ename:str # The attribute cannot be call "name", so we use ename here age:int job:str def __init__(self, ename,age,job): self.ename=ename self.age =age self.job =job def __repr__(self) -> str: return "("+self.ename+","+self.age+","+self.job+")" class State(pc.State): members:list[Member] = [ Member("John",20,"Developer"), Member("May",23,"Design"), Member("Milo",18,"Teacher"), Member("Aloha",48,"CEO"), ] def delete_member(self,member_name:str): del_idx:int = -1 for i in range(len(self.members)): if(self.members[i].ename == member_name): del_idx = i break if(del_idx > -1): del self.members[del_idx] return def index() -> pc.Component: return pc.center( pc.vstack( pc.vstack( pc.hstack( pc.heading("Members"), ), pc.table_container( pc.table( pc.thead( pc.tr( pc.th("Name"), pc.th("Age"), pc.th("Job"), pc.th("Action"), ) ), pc.tbody( pc.foreach( State.members, lambda member: pc.tr( pc.td(member.ename), pc.td(member.age), pc.td(member.job), pc.td( pc.button( "Delete", on_click=lambda: State.delete_member(member.ename), bg="red", color="white", ) ), ) ) ), ), bg="#F7FAFC ", border="1px solid #ddd", border_radius="25px", ), align_items="left", padding_top="7em", ), ), padding="1em", ) # Add state and page to the app. app = pc.App(state=State) app.add_page(index, title="table container") app.compile()  (1) You need to define a class. In my case, it is Member(pc.Model, table=True). You must add (pc.Model, table=True) Can we use [list[list[str]] to represent the self.members? The answer is NO. That's we we use members:list[Member] in State but not members:list[list[str]]. (2) When your code satisfies the above condition 1, then we can use pc.foreach() to generate table's data in UI. pc.foreach( State.members, lambda member: pc.tr( pc.td(member.ename), pc.td(member.age), pc.td(member.job), pc.td( pc.button( "Delete", on_click=lambda: State.delete_member(member.ename), bg="red", color="white", ) ), ) )
Question: I have the following code: from transformers import T5Tokenizer, T5ForConditionalGeneration tokenizer = T5Tokenizer.from_pretrained("t5-small") model = T5ForConditionalGeneration.from_pretrained("t5-small") input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids sequence_ids = model.generate(input_ids) sequences = tokenizer.batch_decode(sequence_ids) sequences  Currently it produces this: ['<pad><extra_id_0> park offers<extra_id_1> the<extra_id_2> park.
']  Is there a way to prevent the generator to produce certain words (e.g. stopwords = ["park", "offer"])?

Answer: after looking at the docs found out there is a bad_words_ids parameter that you can pass in the generate() given a bad word list you can create the id list using tokenizer(bad_words, add_special_tokens=False).input_ids  input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids bad_words = ["park", "offers"] bad_words_ids = tokenizer(bad_words, add_special_tokens=False).input_ids #[[2447], [704]] sequence_ids = model.generate(input_ids, bad_words_ids=bad_words_ids) #tensor([[ 0, 32099, 1061, 19, 3, 9, 710, 1482, 550, 45, 32098, 8, 32097, 1061, 5, 1]]) sequences = tokenizer.batch_decode(sequence_ids) print(sequences) #['<pad><extra_id_0> Park is a short walk away from<extra_id_1> the<extra_id_2> Park.
']  Notice how the word "Park" is appearing now. This is because the tokenizer identifies park (id 2447) and Park (id 1061) as 2 different tokens. This may depend on the tokenizer you use (there are case-insensitive tokenizers). If you don't want this to happen you can add Park into the bad word list as well. Colab demo
Question: Using type(z1[0]) I get jaxlib.xla_extension.ArrayImpl. Printing z1[0] I get Array(0.71530414, dtype=float32). How can I get the actual number 0.71530414? I tried z1[0][0] because z1[0] is a kind of array with a single value, but it gives me an error: IndexError: Too many indices for array: 1 non-None/Ellipsis indices for dim 0.. I tried also a different approach: I searched on the web if it was possible to convert from jaxnumpy array to a python list, but I didn't find an answer. Can someone help me to get the value inside a jaxlib.xla_extension.ArrayImpl object?

Answer: You can use float(x[0]) to convert x[0] to a Python float: In [1]: import jax.numpy as jnp In [2]: x = jnp.array([0.71530414]) In [3]: x Out[3]: Array([0.71530414], dtype=float32) In [4]: x[0] Out[4]: Array(0.71530414, dtype=float32) In [5]: float(x[0]) Out[5]: 0.7153041362762451  If you're interested in converting the entire JAX array to a list of Python floats, you can use the tolist() method: In [6]: x.tolist() Out[6]: [0.7153041362762451]
Question: I have a remote sensing photo that has bright non continuous vertical streaks or stripes as in the pic below, my question is there a way to remove them using python and opencv or any other ip library? ,

Answer: You could just do a 7x1 median filter on the image: Input:  After 7x1 median filter:
Question: I have a pandas dataframe with 3 columns. Say it looks like this: test_df = pd.DataFrame({ 'key1': [1, 1, 1, 1, 2, 2, 2], 'key2': ['a', 'b', 'c', 'd', 'e', 'f', 'g'], 'value': ['a-mapped', 'b-mapped', 'c-mapped', 'd-mapped', 'e-mapped', 'f-mapped', 'g-mapped'] })  > test_df key1 key2 value 0 1 a a-mapped 1 1 b b-mapped 2 1 c c-mapped 3 1 d d-mapped 4 2 e e-mapped 5 2 f f-mapped 6 2 g g-mapped  I would like to return a nested dictionary where the first keys are key1 and then then the second key value pair would be {key2: value}. For example, I would want the desired_result to be { 1: {'a': 'a-mapped', 'b': 'b-mapped', 'c': 'c-mapped', 'd': 'd-mapped'}, 2: {'e': 'e-mapped', 'f': 'f-mapped', 'g': 'g-mapped'} }  I can achieve this by using some loops: desired_result = {} for key1 in test_df.key1: desired_result[key1] = {} for idx, row in test_df.iterrows(): desired_result[row.key1][row.key2] = row.value  Is there a more efficient way of doing this?

Answer: Try: out = test_df.groupby("key1").apply(lambda x: dict(zip(x["key2"], x["value"]))).to_dict() print(out)  Prints: { 1: {"a": "a-mapped", "b": "b-mapped", "c": "c-mapped", "d": "d-mapped"}, 2: {"e": "e-mapped", "f": "f-mapped", "g": "g-mapped"}, }
Question: If I have a pandas dataframe with the following columns: id, num, amount. I want to group the dataframe such that all rows in each group have the same id and amount and where each row's value of num has a value that is not more than 10 larger or smaller the next row's value of num. For the same id, if one row to the next does not have the same amount or if the absolute difference between the two num values is more than 10 then it will start a new grouping. Having a row with a different id in the middle does not break a grouping. How can I go about doing this? I have not managed to make a grouping where I'm not looking for matching values (like here where I need it to be close - but not matching). I'm assuming that this would need some custom grouping function but I've been having trouble putting one together Example dataframe:     id amount num     aaa-aaa 130 12   aaa-aaa 130 39   bbb-bbb 270 41   ccc-ccc 130 19   bbb-bbb 270 37   aaa-aaa 130 42   aaa-aaa 380 39     Expected Groups: Group 1:     id amount num     aaa-aaa 130 12     Group 2:     id amount num     aaa-aaa 130 39   aaa-aaa 130 42     Group 3:     id amount num     bbb-bbb 270 41   bbb-bbb 270 37     Group 4:     id amount num     ccc-ccc 130 19     Group 5:     id amount num     aaa-aaa 380 39    

Answer: The logic is not fully clear, but assuming you want to start a new group when there is a gap of more than 10: close = (df.sort_values(by=['amount', 'num']) .groupby('amount') ['num'].diff().abs().gt(10).cumsum() ) for _, g in df.groupby(['amount', close]): print(g, end='\n\n')  Output:  id amount num 0 aaa-aaa 130 12 3 ddd-ddd 130 19 id amount num 1 bbb-bbb 130 39 id amount num 2 ccc-ccc 270 41 4 eee-eee 270 37  how it works: # sort values by amount/sum df.sort_values(by=['amount', 'num']) id amount num 0 aaa-aaa 130 12 3 ccc-ccc 130 19 1 aaa-aaa 130 39 5 aaa-aaa 130 42 4 bbb-bbb 270 37 2 bbb-bbb 270 41 6 aaa-aaa 380 39 # get the absolute successive difference in "num" (df.sort_values(by=['amount', 'num']) .groupby('amount') ['num'].diff().abs() ) 0 NaN 3 7.0 1 20.0 5 3.0 4 NaN 2 4.0 6 NaN Name: num, dtype: float64 # check if it's greater than 10 and cumsum # to create a grouper for groupby [...].gt(10).cumsum() 0 0 3 0 1 1 5 1 4 1 2 1 6 1 Name: num, dtype: int64
Question: I'm trying to write a method for a features pipeline that returns a polars expression. The method should take a column name as a string and an integer number of days. I want to perform a rolling count on that column using a window equal to the number of days. There doesn't seem to be a rolling_count expression, so I attempted to use rolling_sum_by to no avail. def temporal_rolling_count(col: str, days: int) -> pl.Expr: return ( pl.lit(1) .rolling_sum_by(window_size=f"{days}d", by="date_time") .over(col) .fill_null(0) )  I also tried this method, which was closer but still didn't work in all cases def temporal_rolling_count(col: str, days: int) -> pl.Expr: return ( pl.col(col) .cum_count() .over(col, (pl.col("date_time") - pl.col("date_time").min()).dt.days() % days == 0) .fill_null(0) )  Is there anyway to achieve this by returning an expression? Or will I have to act on the DataFrame directly, maybe by using rolling?

Answer: As per the suggestion from @jqurious, by using .clip I was able to achieve the desired outcome without acting on the DataFrame. def temporal_rolling_count(col: str, days: int) -> pl.Expr: return ( pl.col(col).clip(1,1) .rolling_sum(window_size=f"{days}d", by="date_time") .over(col) .fill_null(0) )  EDIT I managed to perform the same thing but only count when a condition was true by doing the following. def temporal_rolling_count(col: str, days: int) -> pl.Expr: return ( pl.when(condition).then(1).otherwise(0) .rolling_sum(window_size=f"{days}d", by="date_time") .over(col) .fill_null(0) )
Question: I just have a newly created Environment in Anaconda (conda 22.9.0 and Python 3.10.10). Then I proceed to install langchain (pip install langchain if I try conda install langchain it does not work). According to the quickstart guide I have to install one model provider so I install openai (pip install openai). Then I enter to the python console and try to load a PDF using the class UnstructuredPDFLoader and I get the following error. What the problem could be? (langchain) C:\Users\user>python Python 3.10.10 | packaged by Anaconda, Inc. | (main, Mar 21 2023, 18:39:17) [MSC v.1916 64 bit (AMD64)] on win32 >>> from langchain.document_loaders import UnstructuredPDFLoader >>> loader = UnstructuredPDFLoader("C:\\<path-to-data>\\data\\name-of-file.pdf") Traceback (most recent call last): File "C:\<path-to-anaconda>\envs\langchain\lib\site-packages\langchain\document_loaders\unstructured.py", line 32, in __init__ import unstructured # noqa:F401 ModuleNotFoundError: No module named 'unstructured' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\<path-to-anaconda>\envs\langchain\lib\site-packages\langchain\document_loaders\unstructured.py", line 90, in __init__ super().__init__(mode=mode, **unstructured_kwargs) File "C:\<path-to-anaconda>\envs\langchain\lib\site-packages\langchain\document_loaders\unstructured.py", line 34, in __init__ raise ValueError( ValueError: unstructured package not found, please install it with `pip install unstructured` 

Answer: Run this pip install unstructured or this pip install "unstructured[local-inference]"
Question: There is a data frame with a model and an item as a column below df = pd.DataFrame({'model':['A','A','A','A','A','A','A','B','B','B','B','B','B','B'], 'item':['aa','ab','ab','ab','ac','ad','ad','ba','ba','ba','bb','bb','bb','bc']})   I want to add a serial column to this data frame, but there are some rules  The serial number is reset when the model(A, B) is changed and starts from zero.  Serial cannot exceed 3. 0, 1, 2 values only After two(2), it starts at zero(0)  In the case of the same item, serial is be the same   what I want is 

Answer: You want pd.factorize on item within each model group (groupby). The reset part is just a modulo away: df['serial'] = df.groupby(['model'])['item'].transform(lambda x: pd.factorize(x)[0]) % 3  Output:  model item serial 0 A aa 0 1 A ab 1 2 A ab 1 3 A ab 1 4 A ac 2 5 A ad 0 6 A ad 0 7 B ba 0 8 B ba 0 9 B ba 0 10 B bb 1 11 B bb 1 12 B bb 1 13 B bc 2