eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm currently in a scenario where I need to install the following: pip install -r https://raw.githubusercontent.com/intro-stat-learning/ISLP_labs/v2.1.3/requirements.txt  What happens is that the library scipy is not available for some reason, resulting in the following error: (test_islp) PS C:\Users\project> pip install -r https://raw.githubusercontent.com/intro-stat-learning/ISLP_labs/v2.1.3/requirements.txt Collecting numpy==1.24.2 (from -r https://raw.githubusercontent.com/intro-stat-learning/ISLP_labs/v2.1.3/requirements.txt (line 1)) ... ... ERROR: No matching distribution found for scipy==1.11.1  Is there a way to simply ignore libraries that return such error, and still go on with the installation of the remaining libraries? I tried using --ignore-installed but it still leads to the same error The quick and dirty solution would be to simply copy and paste the requirements into a text file, delete scipy and perform pip install again. That is not ideal for me because in a scenario with hundreds of libraries and dozens of potential missing distributions it could lead to tedious work.

Answer: You can use this simple python script to loop through all the packages in the requirements.txt and install them. As far as I can tell, this works on Linux and Windows, not sure about Mac. import subprocess def install_packages(requirements_file): with open(requirements_file, 'r') as file: packages = file.readlines() for package in packages: package = package.strip() try: subprocess.check_call(['pip', 'install', package]) except subprocess.CalledProcessError as e: print(f"Error installing package: {package}") if __name__ == '__main__': requirements_file = 'requirements.txt' install_packages(requirements_file)
Question: I have the following DataFrame with MultiIndex columns (the same applies to MultiIndex rows):  import pandas as pd df = pd.DataFrame(columns=pd.MultiIndex.from_product([['A','B'],[1,2,3,4]]), data=[[0,1,2,3,4,5,6,7],[10,11,12,13,14,15,16,17], [20,21,22,23,24,25,26,27]])  Now I want to select the following columns into a new DataFrame: from the A group, elements at index [1,2,4], AND from the B group, elements at index [1,3]. So my new DataFrame would have 5 columns. I can easily make any of the two selections separately using .loc:  df_grA = df.loc[:,('A',(1,2,4))] df_grB = df.loc[:,('B',(1,3))]  But I cannot find a way to achieve what I want. The only way that I can think of is to concat the two pieces together like that:  df_selection = pd.concat([df_grA,df_grB],axis=1)  This works, but it's clunky. I can't believe there's not a more convenient way.

Answer: Another idea with product: from itertools import product print (df.loc[:, list(product('A',(1,2,4))) + list(product('B',(1,2,3)))]) A B 1 2 4 1 2 3 0 0 1 3 4 5 6 1 10 11 13 14 15 16 2 20 21 23 24 25 26
Question: I would like to extract event pair (start and end marked by + and -). but the pairs maybe not match which means start happen two times then followed the end event. In below example, event B start happed 2 times, so I wish it output a mismatched pair with nil in the end event not found. import re import pandas as pd data = """ 00:00:00 +running A dummy data 00:00:01 -running 00:00:02 +running B dummy data 00:00:03 +running B 00:00:04 -running 00:00:05 +running C dummy data 00:00:06 -running 00:00:07 +running D 10:00:08 -running """ m = re.findall(r"(\d+:\d+:\d+) \+running (\w+).*?(\d+:\d+:\d+) \-running",data,re.DOTALL) print(len(m)) df = pd.DataFrame(m,columns=['ts1','name','ts2']) print(df)  Current output:  ts1 name ts2 0 00:00:00 A 00:00:01 1 00:00:02 B 00:00:04 2 00:00:05 C 00:00:06 3 00:00:07 D 10:00:08  Expected:  ts1 name ts2 0 00:00:00 A 00:00:01 1 00:00:02 B NA 2 00:00:03 B 00:00:04 3 00:00:05 C 00:00:06 4 00:00:07 D 10:00:08  What's proper way to get such results in python? I do not care about if use findall or not.

Answer: You can just modify your regex to have an optional trailing part: data = """ 00:00:00 +running A dummy data 00:00:01 -running 00:00:02 +running B dummy data 00:00:03 +running B 00:00:04 -running 00:00:05 +running C dummy data 00:00:06 -running 00:00:07 +running D 10:00:08 -running """ m = re.findall(r"(\d+:\d+:\d+) \+running (\w+)(?:\n(\d+:\d+:\d+) \-running)?", data, re.DOTALL) df = pd.DataFrame(m, columns=['ts1','name','ts2']).replace('', None)  NB. replacing .*? by \n. Output:  ts1 name ts2 0 00:00:00 A 00:00:01 1 00:00:02 B None 2 00:00:03 B 00:00:04 3 00:00:05 C 00:00:06 4 00:00:07 D 10:00:08  regex demo Handling dummy data If you assume there could be arbitrary rows of data you could filter them out if they don't match the expected pattern: data = """ 00:00:00 +running A dummy data 00:00:01 -running 00:00:02 +running B dummy data 00:00:03 +running B 00:00:04 -running 00:00:05 +running C dummy data 00:00:06 -running 00:00:07 +running D 10:00:08 -running """ m = re.findall(r"(\d+:\d+:\d+) \+running (\w+)(?:\s*(\d+:\d+:\d+) \-running)?", '\n'.join(x for x in data.splitlines() if re.match(r'\d+:\d+:\d+ [-+]running', x)), re.DOTALL) df = pd.DataFrame(m,columns=['ts1','name','ts2']).replace('', None)  Or using a negative lookahead to handle the dummy part: m = re.findall(r"(\d+:\d+:\d+) \+running (\w+)(?:(?:.(?!\d+:\d+:\d+))*\n(\d+:\d+:\d+) \-running)?", data, re.DOTALL) df = pd.DataFrame(m,columns=['ts1','name','ts2']).replace('', None)  regex demo Output:  ts1 name ts2 0 00:00:00 A 00:00:01 1 00:00:02 B None 2 00:00:03 B 00:00:04 3 00:00:05 C 00:00:06 4 00:00:07 D 10:00:08
Question: I am executing the below two code snippets to calculate the cosine similarity of two vectors where the vectors are the same for both executions and the code for the second one is mainly the code SciPy is running (see scipy cosine implementation). The thing is that when calling SciPy it is running slightly faster (~0.55ms vs ~0.69ms) and I don't understand why, as my implementation is like the one from SciPy removing some checks, which if something I would expect to make it faster. Why is SciPy's function faster?  import time import math import numpy as np from scipy.spatial import distance SIZE = 6400000 EXECUTIONS = 10000 path = "" # From https://github.com/joseprupi/cosine-similarity-comparison/blob/master/tools/vectors.csv file_data = np.genfromtxt(path, delimiter=',') A,B = np.moveaxis(file_data, 1, 0).astype('f') accum = 0 start_time = time.time() for _ in range(EXECUTIONS): cos_sim = distance.cosine(A,B) print(" %s ms" % (((time.time() - start_time) * 1000)/EXECUTIONS)) cos_sim_scipy = cos_sim def cosine(u, v, w=None): uv = np.dot(u, v) uu = np.dot(u, u) vv = np.dot(v, v) dist = 1.0 - uv / math.sqrt(uu * vv) # Clip the result to avoid rounding error return np.clip(dist, 0.0, 2.0) accum = 0 start_time = time.time() for _ in range(EXECUTIONS): cos_sim = cosine(A,B) print(" %s ms" % (((time.time() - start_time) * 1000)/EXECUTIONS)) cos_sim_manual = cos_sim print(np.isclose(cos_sim_scipy, cos_sim_manual))  EDIT: The code to generate A and B is below and the exact files I am using can be found at: https://github.com/joseprupi/cosine-similarity-comparison/blob/master/tools/vectors.csv def generate_random_vector(size): """ Generate 2 random vectors with the provided size and save them in a text file """ A = np.random.normal(loc=1.5, size=(size,)) B = np.random.normal(loc=-1.5, scale=2.0, size=(size,)) vectors = np.stack([A, B], axis=1) np.savetxt('vectors.csv', vectors, fmt='%f,%f') generate_random_vector(640000)  Setup:  AMD Ryzen 9 3900X 12-Core Processor 64GB RAM Debian 12 Python 3.11.2 scipy 1.13.0 numpy 1.26.4 

Answer: It seems, scipy does at the beginning of correlation() function this which practically means: u = np.asarray(u, dtype=None, order="c") v = np.asarray(v, dtype=None, order="c")  This ensures that the arrays are C_CONTIGUOUS (you can check this by printing u.flags and/or v.flags) I presume numpy uses different implementations of np.dot for contiguos/non-contiguos arrays.  If you change your function to: def cosine(u, v, w=None): u = np.asarray(u, dtype=None, order="c") # <-- Ensure C_CONTIGUOUS True v = np.asarray(v, dtype=None, order="c") # <-- detto. uv = np.dot(u, v) uu = np.dot(u, u) vv = np.dot(v, v) dist = 1.0 - uv / math.sqrt(uu * vv) # Clip the result to avoid rounding error return np.clip(dist, 0.0, 2.0)  I get the same results 0.45ms vs 0.45ms on my AMD 5700x.
Question: I have a list of colors that created manually; colmap = ['green', 'red', 'red', 'red', 'green']  and also have a pandas dataframe df with same length of my colmap Percent 0 -0.5 1 0.2 2 0.8 3 -0.3 4 0.6  I want to apply the colors to only Percent column, df.style.applymap doesn't work because it requires a function to apply styles. I couldn't figure out how to apply without a lambda function. Thanks in advance

Answer: You could pass a DataFrame to style. The callable doesn't need to use the input data: df.style.apply( lambda x: pd.DataFrame( {'Percent': [f'background-color:{x}' for x in colmap]}, index=x.index ), axis=None, )  NB. since the styling DataFrame should be aligned, you still should pass the index. Output:  Example with another column:  mapping to several columns If you want to use different mappers for different colors: other_colors = ['background-color:white', 'background-color:pink', 'background-color:teal', 'background-color:pink', 'background-color:white'] df.style.apply( lambda x: pd.DataFrame( {'Percent': [f'background-color:{x}' for x in colmap], 'other': other_colors, }, index=x.index ).reindex(columns=x.columns), axis=None, )  Or even: colmap = ['green', 'red', 'red', 'red', 'green'] other_colors = ['white', 'pink', 'teal', 'pink', 'white'] df.style.apply( lambda x: pd.DataFrame( {'Percent': colmap, 'other': other_colors }, index=x.index ).radd('background-color:') .reindex(columns=x.columns), axis=None, )  Or using a variant of @antoine's approach, with axis=0 and a dictionary: colors = {'Percent': ['green', 'red', 'red', 'red', 'green'], 'other': ['white', 'pink', 'teal', 'pink', 'white'] } def lst_to_property(lst): if lst is not None: return [f'background-color:{x}' for x in lst] df_.style.apply(lambda c: lst_to_property(colors.get(c.name, None)), axis=0)
Question: Assume I have the following Polars DataFrame: all_items = pl.DataFrame( { "ISO_codes": ["fin", "nor", "eng", "eng", "swe"], "ISO_codes1": ["fin", "nor", "eng", "eng", "eng"], "ISO_codes2": ["fin", "ice", "eng", "eng", "eng"], "OtherColumn": ["1", "2", "3", "4", "5"], })  How can I implement a method like check_for_equality along the lines of def check_for_equality(all_items, columns_to_check_for_equality): return all_items.with_columns( pl.col_equals(columns_to_check_for_equality).alias("ISO_EQUALS") )  So that when I call it: columns_to_check_for_equality = ["ISO_codes", "ISO_codes1", "ISO_codes2"] resulting_df = check_for_equality(all_items, columns_to_check_for_equality)  I achieve the following: resulting_df == pl.DataFrame( { "ISO_codes": ["fin", "nor", "eng", "eng", "swe"], "ISO_codes1": ["fin", "nor", "eng", "eng", "eng"], "ISO_codes2": ["fin", "ice", "eng", "eng", "eng"], "OtherColumn": ["1", "2", "3", "4", "5"], "ISO_EQUALS": [True, False, True, True, False], })  Note that I do not "know" the column names when doing the actual check, and the number of columns can vary between calls. Is there something like "col_equals" in the Polars API?

Answer: pl.n_unique_horizontal would probably be the answer, but it has not yet been added.  https://github.com/pola-rs/polars/issues/9966  You can use the List API: df.with_columns(ISO_EQUALS = pl.concat_list(columns_to_check_for_equality).list.n_unique() == 1 )  shape: (5, 5) ┌───────────┬────────────┬────────────┬─────────────┬────────────┐ │ ISO_codes ┆ ISO_codes1 ┆ ISO_codes2 ┆ OtherColumn ┆ ISO_EQUALS │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ str ┆ bool │ ╞═══════════╪════════════╪════════════╪═════════════╪════════════╡ │ fin ┆ fin ┆ fin ┆ 1 ┆ true │ │ nor ┆ nor ┆ ice ┆ 2 ┆ false │ │ eng ┆ eng ┆ eng ┆ 3 ┆ true │ │ eng ┆ eng ┆ eng ┆ 4 ┆ true │ │ swe ┆ eng ┆ eng ┆ 5 ┆ false │ └───────────┴────────────┴────────────┴─────────────┴────────────┘
Question: I have a function which will increment a number with time. This is given to a process; but it doesn't increment while the process runs. I want the incremented value of my global variable at regular intervals. But, I am not seeing the increment happening. Here is what I tried: from multiprocessing import Process from time import sleep x = 0; def increment(): global x; while(1): x +=1 sleep(1) process_1 = Process(target=increment) process_1.start() for i in range(0,10): print(x); sleep(2)  I am expecting that it will print 1 3 5 7 ...  However, it remains at 0. What am I missing here?

Answer: Note that each process is a separate object(). You should pass the vars in the process, to be locally saved in the memory. from multiprocessing import Process, Value from time import sleep def increment(x): while True: x.value += 1 sleep(1) if __name__ == '__main__': x = Value('i', 1) process_1 = Process(target=increment, args=(x,)) process_1.start() for i in range(10): print(f'{i}: {x.value}') sleep(2)  Prints 0: 1 1: 3 2: 5 3: 7 4: 9 5: 11 6: 13 7: 15 8: 17 9: 19
Question: I have a class decorator p() that can be used only on classes that inherite from some abstract class P. So anytime someone uses p it should be written as follows: @p() class Child(P): ...  Is there any way to automatically inherite P using the decorator p() so it would simply be written as follows: @p() class Child: ... 

Answer: As I commented, it's tricky to apply a base class using a decorator the way you asked. However, it is a lot easier if you try an alternative approach, where the decorator's behavior is incorporated into the base class, rather than the other way around. Here's an example, that prints out a message before calling the child class's __init__ method: from functools import wraps class Base: def __init_subclass__(cls): original_init = cls.__init__ @wraps(cls.__init__) def wrapper(self, *args): print("foo") original_init(self, *args) cls.__init__ = wrapper class Derived(Base): def __init__(self): print("bar")  You can customize the wrapper function's behavior to do whatever your decorator currently does (rather than just printing a message). And of course, the base class can have whatever other behavior you want the derived class to inherit, rather than just the __init_subclass__ machinery that enables changing the subclass's __init__ method.
Question: Getting different output for the same logic? Was trying to learn async/await in python. Output for first Snippet : <class 'dict'> Output for second Snippet : <class 'list'>  Snippet 1 : without async import urllib.request import json def get_response(user_url): resp = urllib.request.urlopen(user_url).read() resp_json = json.loads(resp) return resp_json['results'][0] def make_request(user_url): json_response = get_response(user_url) print(type(json_response)) USER_URL = 'https://randomuser.me/api' make_request(USER_URL)  Snippet 2 : with async import urllib.request import json import asyncio async def get_response(user_url): resp = urllib.request.urlopen(user_url).read() resp_json = json.loads(resp) return resp_json['results'][0] async def make_request(user_url): json_response = await asyncio.gather(get_response(user_url)) print(type(json_response)) USER_URL = 'https://randomuser.me/api' asyncio.run(make_request(USER_URL))  Why are we getting different output for the same logic in code? Expectation is to get dict data type for both. Using Python 3.10

Answer: That <class 'list'> is the return value of the asyncio.gather().  If all awaitables are completed successfully, the result is an aggregate list of returned values. You'll then need to iterate through the list to access the result of the awaitables.  If you only have that single URL, await get_response(user_url) is enough. asyncio.gather() is for executing multiple awaitables concurrently and wait for them to finished.
Question: the resulting dataframe will sum the value of matching index, but if an index is missing in one dataframe but present in the other, it will just carry them over instead of removing them. for example df1 = data index1 4.2 index2 5.0 index3 3.0  df2 = data index2 1.0 index3 7.0 index4 2.0  result = data index1 4.2 ( carry from df1) index2 6.0 (5.0 + 1.0) index3 10.0 (3.0 + 7.0) index4 2.0 ( carry from df2)  I tried using + and add(), also concatenate, they either remove the index or fill it with na if the index is missing from one of the dataframe

Answer: You can try pd.concat and sum along rows: out = pd.concat([df1, df2], axis=1).sum(axis=1).to_frame(name="data") print(out)  Prints:  data index1 4.2 index2 6.0 index3 10.0 index4 2.0   OR: using .merge: out = ( df1.merge(df2, left_index=True, right_index=True, how="outer") .sum(axis=1) .to_frame(name="data") ) print(out)
Question: My DataFrames are: import pandas as pd df_1 = pd.DataFrame( { 'a': [10, 12, 14, 20, 25, 30, 42, 50, 80] } ) df_2 = pd.DataFrame( { 'start': [9, 19], 'end': [26, 50], 'label': ['a', 'b'] } )  Expected output: Adding column label to df_1: a label 10 a 12 a 14 a 20 a 25 a 20 b 25 b 30 b 42 b 50 b  df_2 defines the ranges of labels. So for example, the first row of df_2 start of the range is 9 and the end is 22. Now I want to slice df_1 based on start and end and give this label to the slice. Note that start is exlusive and end is inclusive. And my labels ranges are overlapping. These are my attempts. The first one works but I am not sure if it is the best. # attempt_1 dfc = pd.DataFrame([]) for idx, row in df_2.iterrows(): start = row['start'] end = row['end'] label = row['label'] df_slice = df_1.loc[df_1.a.between(start, end, inclusive='right')] df_slice['label'] = label dfc = pd.concat([df_slice, dfc], ignore_index=True) ## attempt 2 idx = pd.IntervalIndex.from_arrays(df_2['start'], df_2['end'], closed='both') label = df_2.iloc[idx.get_indexer(df_1.a), 'label'] df_1['label'] = label.to_numpy() 

Answer: I'd try to concatenate generated labeled ranges using pandas.concat this way: template = df_1.set_index('a') ranges = df_2.values output = pd.concat( template.loc[start:end].assign(label=label) for start, end, label in ranges ).reset_index()  It's close to your solution with two major differences:  All iterations are collapsed into an inner generator. We use df_1['a'] as an index, which is implied by its nature.
Question: I was attempting the word search problem here on leetcode. Leetcode seems to give me an error for the two test cases: board = [["a","a"]], word = "aa" and for : board = [["a"]], word = "a" But the following code seems to work fine on Google colab for both of these test cases. Could someone be able to please pinpoint the exact cause? Likely, it is the result of the difference in Python versions. But what exactly is a loss for me! Thanks in advance! class Solution: def exist2(self, board: List[List[str]], word: str, current_row, current_col,\ match_index=0,seen_cells=[]) -> bool: if match_index==len(word): return True else: for i in range(-1,2): for j in range(-1,2): if current_row+i>=0 and current_col+j>=0 and current_row+i<len(board)\ and current_col+j<len(board[0]) and\ board[current_row+i][current_col+j]==word[match_index] and\ (current_row+i,current_col+j) not in seen_cells\ and i+j!=-2 and i+j!=2: match_index+=1 seen_cells.append((current_row+i,current_col+j)) if self.exist2(board, word, current_row+i, current_col+j,\ match_index,seen_cells): return True else: seen_cells.remove((current_row+i,current_col+j)) current_row,current_col=seen_cells[-1] return False def exist(self, board: List[List[str]], word: str, current_row=0, current_col=0,\ match_index=0,seen_cells=[]) -> bool: start_index=[] for i in range(len(board)): for j in range(len(board[0])): if board[i][j]==word[0]: start_index.append((i,j)) for ele in start_index: if self.exist2(board, word, ele[0],ele[1]): return True return False def main() print(exist([["a","a"]],'a')) 

Answer: There are multiple issues with your code. First, you are relying on the default argument of seen_cells, but there is only one instance of that list, which is used across all calls to exist2. Multiple test cases will share the same state, which causes problems when there are tuples in seen_cells from previous calls of exist2. You are also incrementing match_index in every iteration of the inner loop in exist2 even though each iteration should be independent. You should instead pass match_index + 1 to the recursive call of exist2. Furthermore, you have allowed diagonal moves by only checking i+j!=-2 and i+j!=2. For instance, the move (-1, 1) is allowed since -1 + 1 = 0 is neither equal to -2 nor 2. It would be correct to check that exactly one of i and j is non-zero. This can be done concisely with ((not i) ^ (not j)), though you can also write it out more explicitly. In addition, you do not add the first cell in the path to seen_paths. You can fix this off-by-one error by checking for the match right at the start of exist2 and remove from seen_cells right before the return. Note that the line current_row,current_col=seen_cells[-1] does not make much sense and can cause IndexErrors; remove it. Finally, this solution is too slow because checking for existence in a list is a linear time operation. Instead, use a set or a boolean matrix to mark which cells have already been passed through.
Question: I am making a Math-thingy that requires numbers to be drawn to the screen right aligned. I'm using this function to draw text: def drawText(text, font, text_col, x, y): img = font.render(text, True, text_col) screen.blit(img, (x, y))  How do I draw the text right aligned instead of left aligned? I tried to use Python's .rjust() function for this, but I'm not using a monospaced font. Is there anyway to possibly either use .rjust() to right align it or possibly something else?

Answer: The pygame.Rect object has a set of virtual attributes that can be used to define the position of surfaces and text. Align the text to its right by setting topright: def drawText(text, font, text_col, x, y): img = font.render(text, True, text_col) rect = img.get_rect(topright = (x, y)) screen.blit(img, rect)   Minimal example  import pygame from pygame.locals import * pygame.init() screen = pygame.display.set_mode((500, 300)) clock = pygame.time.Clock() font = pygame.font.SysFont(None, 32) count = 0 run = True while run: for event in pygame.event.get(): if event.type == pygame.QUIT: run = False count += 1 text_surf = font.render(str(count), True, "black") rect = screen.get_rect().inflate(-110, -110) screen.fill("white") pygame.draw.rect(screen, "red", screen.get_rect().inflate(-100, -100), 5) screen.blit(text_surf, text_surf.get_rect(topleft = rect.topleft)) screen.blit(text_surf, text_surf.get_rect(midtop = rect.midtop)) screen.blit(text_surf, text_surf.get_rect(topright = rect.topright)) screen.blit(text_surf, text_surf.get_rect(midleft = rect.midleft)) screen.blit(text_surf, text_surf.get_rect(center = rect.center)) screen.blit(text_surf, text_surf.get_rect(midright = rect.midright)) screen.blit(text_surf, text_surf.get_rect(bottomleft = rect.bottomleft)) screen.blit(text_surf, text_surf.get_rect(midbottom = rect.midbottom)) screen.blit(text_surf, text_surf.get_rect(bottomright = rect.bottomright)) pygame.display.flip() clock.tick(100) pygame.quit() exit()