eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to render a gym env in test but not in learning. Here is my code: import gymnasium as gym import numpy as np env = gym.make('FrozenLake-v1') # initialize Q table Q = np.zeros([env.observation_space.n, env.action_space.n]) print(Q) # parameter lr = 0.8 gamma = 0.95 num_episodes = 2000 # learning for i in range(num_episodes): state = env.reset()[0] done = False while not done: # epsilon-greedy policy if np.random.uniform(0, 1) < 0.5: action = env.action_space.sample() else: action = np.argmax(Q[state,:]) next_state, reward, done, _, _ = env.step(action) Q[state, action] = (1 - lr) * Q[state, action] + \ lr * (reward + gamma * np.max(Q[next_state, :])) state = next_state print(Q) # test state = env.reset()[0] done = False while not done: action = np.argmax(Q[state, :]) next_state, reward, done, _, _ = env.step(action) state = next_state env.render()  It doesn't render and give warning: WARN: You are calling render method without specifying any render mode. You can specify the render_mode at initialization, e.g. gym.make("FrozenLake-v1", render_mode="rgb_array") If I specify the render_mode to 'human', it will render both in learning and test, which I don't want. How should I do?

Answer: You can just recreate a new environment specifying the render mode. import gymnasium as gym import numpy as np env_train = gym.make('FrozenLake-v1') # initialize Q table Q = np.zeros([env_train.observation_space.n, env_train.action_space.n]) print(Q) # parameter lr = 0.8 gamma = 0.95 num_episodes = 2000 # learning for i in range(num_episodes): state = env_train.reset()[0] done = False while not done: # epsilon-greedy policy if np.random.uniform(0, 1) < 0.5: action = env_train.action_space.sample() else: action = np.argmax(Q[state,:]) next_state, reward, done, _, _ = env_train.step(action) Q[state, action] = (1 - lr) * Q[state, action] + \ lr * (reward + gamma * np.max(Q[next_state, :])) state = next_state print(Q) env_train.close() # test env_test = gym.make('FrozenLake-v1', render_mode="human") state = env_test.reset()[0] done = False while not done: action = np.argmax(Q[state, :]) next_state, reward, done, _, _ = env_test.step(action) state = next_state env_test.render() env_test.close()
Question: I have a dataframe like this. I want to sum up special rows containing exact characters that matched my targets.  Ko_EC FPKM count 0 1.1.1.1 16.7 1 1 1.1.1.15 30.0 7 2 4.2.1.128 40.5 9 3 4.2.1.12 57.0 10 4 3.2.1.1 1.1.1.1 22.0 4  Here are my dataframe and my targets. # coding=utf-8 import pandas as pd import numpy as np ######### classes = [('1.1.1.1', 16.7, 1), ('1.1.1.15', 30, 7), ('4.2.1.128', 40.5, 9), ('4.2.1.12', 57, 10), ('3.2.1.1 1.1.1.1', 22, 4)] labels = ['Ko_EC','FPKM', 'count'] alls = pd.DataFrame.from_records(classes, columns=labels) target_list = ['1.1.1.1', '4.2.1.128', '4.2.1.12; 1.1.1.15; 3.2.1.1', '1.1.1.15']  I want to sum up the alls['Ko_EC'] rows containing exact characters that matched the unique target_list. Based on the answer to a previous question, I used this code: result = pd.DataFrame() for target in target_list: mask = alls.apply(lambda x: any([cls in target for cls in x['Ko_EC'].split(' ')]), axis=1) target_sum = alls.loc[mask,["FPKM", 'count']].sum().reset_index().rename(columns={0:target}).iloc[:,1:] result = pd.concat([result,target_sum], axis=1) #concat as result result = result.T result.columns = ['FPKM', 'Count']  It results like:  FPKM Count 1.1.1.1 38.7 5.0 4.2.1.128 97.5 19.0 4.2.1.12; 1.1.1.15; 3.2.1.1 125.7 22.0 1.1.1.15 68.7 12.0  What I want is  FPKM Count 1.1.1.1 38.7 5.0 4.2.1.128 40.5 9.0 4.2.1.12; 1.1.1.15; 3.2.1.1 109.0 21.0 1.1.1.15 30.0 7.0  Here, the 4.2.1.128 row FPKM Count only sum that of itself, and 4.2.1.12; 1.1.1.15; 3.2.1.1 rows sum that of 4.2.1.12, 1.1.1.15 and 3.2.1.1 1.1.1.1 in origin alls dataframe, but my code confused the characters 1.1.1.1/1.1.1.15 and 4.2.1.128/4.2.1.12, because of their similarity. Could anyone tell me how to do this and why my code cannot work? Thanks!

Answer: Using a double explode, a merge then groupby.agg: # create a reference from target_list ref = (pd.Series(target_list, name='Ko_EC') .str.split(r';\s*') .explode() .reset_index() ) # merge the exploded "alls" then aggregate out = ( ref.merge(alls.assign(Ko_EC=alls['Ko_EC'].str.split()).explode('Ko_EC'), how='left') .groupby('index') .agg({'Ko_EC': '; '.join, 'FPKM': 'sum', 'count': 'sum'}) ) print(out)  NB. You can use a single pipeline, I split it in two for clarity. Output:  Ko_EC FPKM count index 0 1.1.1.1; 1.1.1.1 38.7 5 1 4.2.1.128 40.5 9 2 4.2.1.12; 1.1.1.15; 3.2.1.1 109.0 21 3 1.1.1.15 30.0 7
Question: I am tring to run subprocess.run(["huggingface-cli", "login", "--token", TOKEN]) in a Jupyter notebook, which works on Mac but gets the following error on Ubuntu. I checked that pip install huggingface_hub has been executed. subprocess.run(["git", "lfs", "install"]) works. !huggingface-cli login in the jupyter cell gets the error /bin/bash: huggingface-cli: command not found Anyone can help?

Answer: Your huggingface_hub is not in your path env variable for your Ubuntu system, it is not the same between your jupyter and your terminal session here what you can do, get the path of the executable pip show huggingface_hub | grep Location then update the path in your jupyter notebook, like for example: import os import sys import subprocess huggingface_bin_path = "/home/user/.local/bin" os.environ["PATH"] = f"{huggingface_bin_path}:{os.environ['PATH']}" subprocess.run(["huggingface-cli", "login", "--token", TOKEN])  replace /home/user/.local/bin and TOKEN with the correct one, then it should work
Question: I'm trying to generate a 2-dim coordinates matrix using python. I'm using x=np.linespace(min, max, step) y=np.linespace(min, max, step) X, Y = np.meshgrid(x, y)  to generate x and y coordinates, where X like: [[0. 1. 2. 3. 4.] [0. 1. 2. 3. 4.] [0. 1. 2. 3. 4.] [0. 1. 2. 3. 4.] [0. 1. 2. 3. 4.] [0. 1. 2. 3. 4.] [0. 1. 2. 3. 4.] [0. 1. 2. 3. 4.]]  and Y like: [[-2. -2. -2. -2. -2.] [-1. -1. -1. -1. -1.] [ 0. 0. 0. 0. 0.] [ 1. 1. 1. 1. 1.] [ 2. 2. 2. 2. 2.] [ 3. 3. 3. 3. 3.] [ 4. 4. 4. 4. 4.] [ 5. 5. 5. 5. 5.]]  I want to get: [[[0, -2] [0, -1] [0, 0] [0, 1] [0, 2]] [[1, -2] [1, -1] [1, 0] [1, 1] [1, 2]] [[2, -2] [2, -1] [2, 0] [2, 1] [2, 2]] [[3, -2] [3, -1] [3, 0] [3, 1] [3, 2]] [[4, -2] [4, -1] [4, 0] [4, 1] [4, 2]]]  (or its horizontal mirror) How to do that?

Answer: You can implement something like this: #!/usr/bin/env ipython # --------------------------- import numpy as np x0,x1 = -2, 2 y0,y1 = 0,4 x=np.arange(x0,x1, 1) y=np.arange(y0,y1, 1) X, Y = np.meshgrid(x, y) ny,nx = np.shape(X) # ----------------------------------------------------------- ans = [[[X[jj,ii],Y[jj,ii]] for ii in range(nx) ] for jj in range(ny)]  I switched to np.arange instead of np.linspace.
Question: I'm working with a huge .7z file that I need to process line by line. First I tried py7zr, but it only works by first decompressing the whole file into an object. This runs out of memory. Then libarchive is able to read block by block, but there's no straightforward way of splitting these binary blocks into lines. What can I do? Related questions I researched first:  How to read contents of 7z file using python: The answers only decompress the whole file. How to read from a text file compressed with 7z?: Seeks Python 2.7 answers. Python: How can I read a line from a compressed 7z file in Python?: Focuses on a single line, no accepted answer - only answer posted 7 years ago.  I'm looking for ways to improve the temporary solution I built myself - posted as an answer here. Thanks!

Answer: This solution goes through all available get_blocks(). If the last line doesn't end in \n, we keep the remaining bytes to be yield on the next block. import libarchive def process(my_file): data = '' with libarchive.file_reader(my_file) as e: for entry in e: for block in entry.get_blocks(): data += block.decode('ISO-8859-1') lines = data.splitlines() if not data.endswith('\n'): data = lines.pop() else: data = '' for line in lines: yield ({'l': line},)
Question: Yesterday this works fine, today i'm getting error on my local machine, colab notebook, even on my VPS. /usr/local/lib/python3.9/dist-packages/pytube/cipher.py in get_throttling_plan(js) 409 match = plan_regex.search(raw_code) 410 --> 411 transform_plan_raw = find_object_from_startpoint(raw_code, match.span()[1] - 1) 412 413 # Steps are either c[x](c[y]) or c[x](c[y],c[z])  from pytube import YouTube def audio_download(video_url): audio_file = YouTube(video_url).streams.filter(only_audio=True).first().download(filename="audio.mp4") return 'ok' Expected to download audio. I even tried to changes in cipher.py file as said in other solution but not working.

Answer: Found a solution. cipher.py Line 411 transform_plan_raw = find_object_from_startpoint(raw_code, match.span()[1] - 1)  to transform_plan_raw = js
Question: In CPython 3.11, the following code returns very large reference counts for some objects. It seems to follow pre-cached objects like integers -5 to 256, but CPython 3.10 does not: Python 3.11.2 (tags/v3.11.2:878ead1, Feb 7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> for i in (-6, -5, 0, 255, 256, 257): ... print(i, sys.getrefcount(i)) ... -6 5 -5 1000000004 0 1000000535 255 1000000010 256 1000000040 257 5  Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> for i in (-6, -5, 0, 255, 256, 257): ... print(i, sys.getrefcount(i)) ... -6 5 -5 6 0 234 255 8 256 26 257 5  PEP 683 - Immortal Objects, Using a Fixed Refcount may be related, but it isn't mentioned in What's New in Python 3.11, nor is a change in sys.getrefcount() documented. Anybody have knowledge about this change?

Answer: This is not related to PEP 683, but it will probably be superseded by PEP 683 once PEP 683 has been implemented. This refcount change was first introduced in a commit on December 13, 2021, which introduced a _PyObject_IMMORTAL_INIT macro: #define _PyObject_IMMORTAL_INIT(type) \ { \ .ob_refcnt = 999999999, \ .ob_type = type, \ }  Before this change, small int refcounts were initialized to 1. This change made it so their refcounts were instead initialized to 999999999. The relevant issue discussion is here, and the pull request for the commit is here. The value of 999999999 seems to have been copied from a fairly arbitrary decision Guido van Rossum made while writing an unrelated tool. Quoting a message by Guido in the issue discussion:  I used 999999999 in deepfreeze.py to signify "immortal object". It has been copied by others (small integers are essentially immortal too). I wasn't too sure that the refcount wouldn't go below zero if the interpreter is repeatedly finalized and reinitialized. Once we have official immortal objects (see PEP-683) we should switch to that.
Question: In my current project, we are using an OpenAPI-to-TypeScript-API generator, that generates automatically typed functions for calling API endpoints via Axios. In Python, we use snake_case for our class properties, while in TypeScript we use camelCase. Using this setup, we have found that the alias property (Field(..., alias="***")) is very helpful, combined with the allow_population_by_field_name = True property. Simple model class example: from pydantic import BaseModel, Field class MyClass(BaseModel): my_property: str = Field(..., alias="myProperty") class Config: allow_population_by_field_name = True if __name__ == "__main__": my_object = MyClass(my_property="banana")  The question is: Why doesn't Pylance understand that I want to be able to write MyClass(my_property="banana")? See screenshot from vscode below:  Is there any way to configure Pylance/Pydantic/vscode to better understand this? "Just living with" this is problematic because having red lines like this in our code promotes a mindset of "red is probably ok", so they lose their power.  Edit May 8th, 2023: We found a way to "circumvent" this problem, by creating a wrapper class, containing an alias_generator. It doesn't seem like pylance picks up on this, and therefore it acts as if it doesn't exist. I'm sure using the system like this isn't something people do a lot, but this helped our team. Also it allows us to not write = Field(..., alias="camelCase") everywhere.

Answer: It seems like this is a known problem as you can see from issue #4936. It has to do with the way dataclass_transform (from PEP 681) handles the alias field attribute. There is currently no elegant solution for this, but there may be with Pydantic v2 at some point. I agree that you should not just passively ignore the warnings. It is always better to explicitly ignore them, if you did everything you could to fix the underlying issue. Therefore I propose this is a clear-cut use case for # type: ignore or more specifically # pyright: ignore as suggested here in the documentation. Sometimes you just hit the limits of the Python typing system. And when you do, explicit ignore-directives are the correct way to deal with that.
Question: I am trying to calculate the percentage of leaf damage using R. I am able to do the image segmentation and area calculation using the pliman package, following the vignettes: https://cran.r-project.org/web/packages/pliman/vignettes/pliman_start.html  I am using the following R code. I put a 1cm square as a reference scale. A sample segmented image can be downloaded here: #...........................Load image........................... library(pliman) img_segm <- pliman::image_import("C:/Users/sando/Desktop/test/imgseg/imgseg.jpg") #.........................Analize leaves......................... layout(t(1:2)) meas <- analyze_objects(img_segm, index = "BIM", marker = "id", invert= F,show_image=T, parallel=T, watershed= T, fill_hull= F, tolerance = 30, object_size = "large") # Measures area in pixels area1 <- get_measures(meas)[,c(1,4)] area1 # Correct the measures using an object in cm real.area <- get_measures(meas, id = 1, area ~ 1)[,c(1,4)] real.area #........................Analize contour......................... # Draw a convex hull around the objects. cont_meas <- analyze_objects(img_segm, watershed = T, marker = "id", show_chull = TRUE,index = "BIM", invert= F,show_image=T, parallel=T, fill_hull= F, tolerance = 30, object_size = "large") # shows the convex hull # Measures area real.area2 <- get_measures(cont_meas, id = 1, area ~ 1 )[,c(1,4)]  However, I can't obtain the area damaged, and I can't use color segmentation because background is white. I would like:  Identify each leaf individually Predict or select the missing leaf edges using some form of convex hull detection or ROI. Calculate the area of damage (in red) and total area (leaf + red).  I know it's possible to do some binary transformation. So I've tried the following code: #### detect contours in red library(imager) img_segm2 <- imager::as.cimg(img_segm) plot(img_segm2) # isoblur, greyscale img_segm3 <- isoblur(grayscale(img_segm2),2) > .60 plot(img_segm3) px <- img_segm3 > 0.1 ct <- imager::contours(px,nlevels=3) plot(px) #Add contour lines purrr::walk(ct,function(v) lines(v$x,v$y,col="red",lwd=1.5))   Is there any method that allows me to obtain this in a semi-automatic way? I know it can be done in ImageJ and some software like leafbyte, or bioleaf, but I would like to be able to analyze these images in R or Python. Thank you for your time.

Answer: Alright, here's a start for you: https://github.com/brockbrownwork/leaves I'll polish up this answer later, but here's the main results: Input:  Output:   Fraying: 3.947% Percentage of non-fray holes: 5.589% Edit: If you have a few examples of the images you're going to use, that would be helpful. Mainly I want to know if convex hulls will accurately wrap around your leaves, otherwise we'll have to use something else. Are you looking for the stats of each individual leaf, or just the average of the whole? You may have to change the variable that represents the minimum area of a leaf in pixels. The hole detection wraps a little bit around the border of the leaf too, will fix that.
Question: I'm trying to solve a problem which reads as follows: A queue of eager single-digits (your input) are waiting to enter an empty room. I allow one digit (from the left) to enter the room each minute. Each time a new digit enters the room I chalk up the median of all the digits currently in the room on the chalkboard. [The median is the middle digit when the digits are arranged in ascending order.]. If there are two median numbers (i.e. two middles) then rather than using the average, I chalk up the lower one of the two. I chalk new digits to the right of existing digits so my chalkboard number keeps getting longer. What number ends up on your chalkboard once all the digits are in the room? Consider the example input: 21423814127333  2 (the leftmost) is allowed into the room where it is the only digit so I write 2 on the chalkboard. 1 is then allowed into the room to join 2. The smaller one of these two is used as the median so I chalk up 1 to the right of 2 on the chalkboard (my number is now 21) 4 now enters the room. The median of 1, 2 and 4 is 2 so I add 2 to my chalkboard (my number is now 212) ...this process continues until the final 3 enters the room ... all the numbers are in the room now which, when sorted, are 1,1,1,2,2,2,3,3,3,3,4,7,8,8. There are two median digits but they are both 3 so I add 3 to my chalkboard and my final number is 21222222222233  My current solution: num = input() new = str(num[0]) whole = [num[0]] for i in range(1, len(num)): whole.append(num[i]) whole.sort() new += whole[i//2] print(new)  The problem is that it takes too long - so it passes 6/10 (hidden) test cases but exceeds the time limit for the other 4. Any help would be greatly appreciated.

Answer: You are repeatedly sorting, with key comparison, so total cost is O(N * N log N), that is, it is at least quadratic.  single-digits (your input) are waiting to enter  The key to this problem is the range limit on input. We know that each input x is in this range: 0 <= x < 10  Use counters. We can easily allocate ten of them. Keep a running count of total number of digits that have been admitted to the room. Each time you have to report a median, compute sum of ordered counters, stopping when you get to half the total count. max_val = 10 counter = {i: 0 for i in range(max_val)} ... assert 0 <= input_val < max_val counter[input_val] += 1 cum_sum = 0 for i in range(max_val): cum_sum += counter[i] ...  Since median is a robust statistic, typically there will be some stability in the median you report, e.g. "2, 1, 2, 2, 2, 2". You can use that to speed the computation still further, by incrementally computing the cumulative sum. It won't change the big-Oh complexity, though, as there's a constant number of counters. We're still looking at O(N), since we have to examine each of the N digits admitted to the room and then report the current median. This does beat the O(N log N) cost of an approach that relies on bisecting an ordered vector.
Question: How would you transform a pandas dataframe composed of successive events: start_time | end_time | value 1671221209 1671234984 2000 1671240425 1671241235 1000 1671289246 1671289600 133 ... ... ...  into timeserie like this:  time | value 1671221209 2000 1671234984 2000 1671234985 0 1671240424 0 1671240425 1000 1671241235 1000 1671241236 0 1671289245 0 1671289246 133 1671289600 133 1671289601 0 ... ... 

Answer: You can use melt function: result = ( df.melt(id_vars='value', var_name='time_type', value_name='time') .drop(columns=['time_type']) .sort_values(by='time') .reset_index(drop=True) ) result = result[['time', 'value']]
Question: When I print a Polars dataframe (e.g., from terminal or within Jupyter notebook), there is a leading string citing the shape of the resultant dataframe. I am looking for the method to not have this printed as part of the result. The simple example would be the following: >>> import polars as pl >>> df = pl.DataFrame() >>> df shape: (0, 0) ┌┐ ╞╡ └┘ >>> print(df) shape: (0, 0) ┌┐ ╞╡ └┘ >>> print(df[0, :]) shape: (0, 0) ┌┐ ╞╡ └┘  My question is how would I print only the dataframe without the lurking "shape: [n, m]" string?

Answer: I found a native way: you could use pl.Config.set_tbl_hide_dataframe_shape. This import polars as pl pl.Config.set_tbl_hide_dataframe_shape(True) df = pl.DataFrame({'col1': range(3), 'col2': ['a', 'b', 'c']}) print(df)  results in ┌──────┬──────┐ │ col1 ┆ col2 │ │ --- ┆ --- │ │ i64 ┆ str │ ╞══════╪══════╡ │ 0 ┆ a │ │ 1 ┆ b │ │ 2 ┆ c │ └──────┴──────┘
Question: for example we have:     a b c     1 1 3   2 4 1   3 2 1     And now using python I'm trying to create this:     a b c max     1 1 3 3c   2 4 1 4b   3 2 1 3a    

Answer: If need match first maximal column join max converted to strings with DataFrame.idxmax for columns names by max: cols = ['a','b','c'] df['max'] = df[cols].max(axis=1).astype(str).str.cat(df[cols].idxmax(axis=1)) print (df) a b c max 0 1 1 3 3c 1 2 4 1 4b 2 3 2 1 3a  If possible multiple max values and need all matched columns use DataFrame.dot trick with separator ,: print (df) a b c 0 3 1 3 1 2 4 1 2 3 2 1 cols = ['a','b','c'] max1 = df[cols].max(axis=1) s = df[cols].eq(max1, axis=0).dot(pd.Index(cols) + ',').str[:-1] df['max'] = max1.astype(str).str.cat(s) print (df) a b c max 0 3 1 3 3a,c 1 2 4 1 4b 2 3 2 1 3a
Question: It is often common to write this type of python code: def load_data(): df_ans = get_data() df_checked = check_data(df_ans) # returns df_ans or raises an error return_dict = format_data(df_ans) return return_dict  One could write the above like this (but it's ugly in my opinion) def load_data(): return format_data(check_data(get_data()))  If this were all pandas code, one could use the .pipe method to do as follows: def load_data(): return get_data().pipe(check_data).pipe(format_data)  However, there seems to be no universal way to "pipe" things in Python, something like: def load_data(): return (get_data() |> check_data |> format_data)  Is there any PEP proposal for this? I could not find one. I've seen some libraries that do operator overloading on | but I don't think that's what I'm looking for - I'm looking for x {operator} y to be the same as y(x) (which of course requires y to be callable)

Answer: I don't think a pipe operator in Python will be approved in the near future. To do that operation there is already the function pipe in the toolz library. It is not a Python built in library but is widely used. In case you are curious the implementation is just: def pipe(data, *funcs): for func in funcs: data = func(data) return data
Question: I want a regular expression, which returns only digits, which are within a word, but I can only find expressions, which returns all digits in a string. I've used this example: text = 'I need this number inside my wor5d, but also this word3 and this 4word, but not this 1 and not this 555.' The following code returns all digits, but I am only interested in ['5', '3', '4']  import re print(re.findall(r'\d+', text)) Any suggestions?

Answer: You can use re.findall(r'(?<=[a-zA-Z])\d+|\d+(?=[a-zA-Z])', text)  This regex will extract all one or more digit chunks that are immediately preceded or followed with an ASCII letter. A fully Unicode version for Python re would look like (?<=[^\W\d_])\d+|\d+(?=[^\W\d_])  where [^\W\d_] matches any Unicode letter. See the regex demo for reference.