eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have two datasets/dataframes df1 and df2, I want to generate df3 by finding the difference between numeric columns of df2 and df1's column_X. #### copy and paste below to generate df1 and df2 import pandas as pd from random import uniform import numpy as np # generation of df1 data = np.random.uniform(15,40, size=(60, 2)) df1 = pd.DataFrame(data, columns=['column_A','column_B']) df1['column_X'] = df1.mean(axis=1) df1 # generation of df2 data = np.random.uniform(10.5,32.8, size=(60, 30)) df2 = pd.DataFrame(data, columns=['column_1','column_2','column_3','column_4','column_5', 'column_6','column_7','column_8','column_9','column_10', 'column_11','column_12','column_13','column_14','column_15', 'column_16','column_17','column_18','column_19','column_20', 'column_21','column_22','column_23','column_24','column_25', 'column_26','column_27','column_28','column_29','column_30',]) df2["Group"] = pd.DataFrame(np.repeat(['A','B','C'], 20, axis=0)) # make "Group" column the first column col = df2.pop('Group') df2.insert(0, 'Group', col) df2  I want to generate df3 by substracting df2's numeric columns (column_1 to column_30) from df1's Column_X while retaining the "Group" column # Step 1: create an empty df3 and then append df2['Group'] df3 = pd.DataFrame() # substract "column_X from each numeric column df3['col_1X_sub'] = df2['column_1'] - df1['column_X'] df3['col_2X_sub'] = df2['column_2'] - df1['column_X'] df3['col_3X_sub'] = df2['column_3'] - df1['column_X'] . . . df3['col_30X_sub'] = df2['column_30'] - df1['column_X']  final df3 should look something like this for all 30 columns 

Answer: Copy and then broadcast-subtract: import pandas as pd import numpy as np np.random.seed(0) a, b = ab = np.random.uniform(low=15, high=40, size=(2, 60)) df1 = pd.DataFrame({ 'column_A': a, 'column_B': b, 'column_X': ab.mean(axis=0), }) df2 = pd.DataFrame( data=np.random.uniform(low=10.5, high=32.8, size=(60, 30)), columns=[f'column_{i}' for i in range(1, 31)], ) df2.insert( loc=0, column='Group', value=np.repeat(['A','B','C'], 20, axis=0), ) df3 = df2.copy() df3.iloc[:, 1:] -= df1['column_X'].values[:, np.newaxis]
Question: I have a plot with a line graph. I need to place a new axis at a point/coordinate and plot a pie chart on the new axis so the pie is centered on the point. import matplotlib.pyplot as plt import numpy as np fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(6, 3)) ax.set_xlim(0,16) ax.set_ylim(8,12) #Plot a sine wave x = np.arange(0, 5*np.pi, 0.1) y = np.sin(x)+10 ax.plot(x, y, color="blue") #Plot a red point at x=12, y=10 ax.plot(12,10,marker="o", color="red") #Add a new axes to the plot #Normalize the points coordinates to range between 0 and 1 x_norm = (12-0)/(16-0) #0.75 y_norm = (10-8)/(12-8) #0.5 #Add an ax at the normalized coordinates left=0.75 bottom=0.5 width=0.1 height=0.1 sub_ax = fig.add_axes(rect=(left, bottom, width, height))   sub_ax.pie((0.2,0.3,0.5))  The pie is centered on the new axis center. I cant figure out the logic to get it centered at the point? 

Answer: You could use the inset_axes method, (based on the last option from this answer) e.g., import matplotlib.pyplot as plt import numpy as np fig, ax = plt.subplots(nrows=1, ncols=1, figsize=(6, 3)) ax.set_xlim(0,16) ax.set_ylim(8,12) #Plot a sine wave x = np.arange(0, 5*np.pi, 0.1) y = np.sin(x)+10 ax.plot(x, y, color="blue") #Plot a red point at x=12, y=10 point = (12, 10) ax.plot(*point, marker="o", color="red") # width and height in data coordinates width = 2 height = 0.5 # subtract half the width/height from the left/bottom positions position = [point[0] - width / 2, point[1] - height / 2, width, height] sub_ax = ax.inset_axes(position, transform=ax.transData) sub_ax.pie((0.2,0.3,0.5))
Question: I have some survey question answers (letters A, B, C) I'd like to plot in a Seaborn histogram, with the frequency of each response letter grouped by gender. import pandas as pd answers = ['A', 'B', 'A', 'B', 'A', 'A', 'B', 'B', 'B', 'A', 'C', 'B', 'A', 'C'] genders = ['M', 'M', 'F', 'M', 'F', 'M', 'F', 'M', 'M', 'F', 'M', 'M', 'F', 'M'] df = pd.DataFrame({'Answer': answers, 'Gender': genders})  I'd like the output to resemble either clustered columns:  or stacked columns:  Seaborn isn't mandatory - it's just what I've been trying for a while. I've been struggling for days since categorical data like mine isn't easy to find in online examples. Any help is tremendously appreciated.

Answer: You can use a simple countplot: import seaborn as sns sns.countplot(df, x='Answer', hue='Gender')  Output:  Another option, without seaborn, would be to crosstab, then plot.bar: pd.crosstab(df['Answer'], df['Gender']).plot.bar()  Output:  Which allows you to easily stack: pd.crosstab(df['Answer'], df['Gender']).plot.bar(stacked=True)  Output:
Question: I would like to define custom sympy functions which cannot be directly evaluated, but which I do want to be able to compute some things about. For example, modular remainders. Here's my code attempt: from sympy import Function class Foo(Function): @classmethod def eval(cls, n): pass def __mod__(self, modulus: int) -> int: n, = self.args return (n + 1) % modulus print(Foo(7) % 3) # 2 print((Foo(7) + 1) % 3) # Mod(Foo(7) + 1, 3) print((Foo(7) + 1) % 3 == 0) # False  So Foo(7) % 3 works as expected, but (Foo(7) + 1) % 3 does not, I wanted sympy to notice that it can compute (a + b) % 3 by computing a%3 and b%3. Is there some way to implement this? Perhaps I need to be more clever and write a new function that walks the symbolic tree of the formula? Or maybe I'm going about this the wrong way? I'm also open to other python libraries (I just guessed that sympy might be a good one to use). In case it helps, my real use-case for this is representing integers that are too large to store in memory and so can only be represented by formulas. But where we can still calculate some properties like modular remainders. See for example https://www.sligocki.com/2022/06/21/bb-6-2-t15.html

Answer: SymPy's symbolic Mod expression type looks for a method called _eval_Mod. You don't need to define __mod__ because Expr.__mod__ already returns Mod(self, other) and Mod will look for the _eval_Mod method: In [2]: from sympy import Function ...: ...: class Foo(Function): ...: @classmethod ...: def eval(cls, n): ...: pass ...: ...: def _eval_Mod(self, modulus: Expr) -> Integer: ...: if not isinstance(modulus, Integer): ...: raise TypeError ...: n, = self.args ...: return (n + 1) % modulus ...: In [3]: Foo(7) % 3 Out[3]: 2 In [4]: (Foo(7) + 1) % 3 Out[4]: 0  That makes your function Foo work as part of an expression rather than just making the operator a % b work for an individual Foo instance. I don't know generally what operations you want besides _eval_Mod but just in case you are not aware of this various polynomial functions in SymPy (factor, cancel etc) can accept a modulus argument that maybe of use to you: In [8]: factor(x**2 + 1) Out[8]: 2 x + 1 In [9]: factor(x**2 + 1, modulus=2) Out[9]: 2 (x + 1)
Question: Python version: (3.9, but the same result with python 3.12) The goal was that another thread modified a dictionary and those modifications to be available at the original thread. import multiprocessing as mp import sys def my_func(result: dict): print(f'variable address at worker: {hex(id(result))}', file=sys.stderr) result["test"] = "test" print(f'{result}', file=sys.stderr) result = {} print(f'variable address at main thread: {hex(id(result))}', file=sys.stderr) my_worker = lambda : my_func(result) # execution at another Thread p = mp.Process(target=my_worker) p.start() p.join() print(f'result at main thread after execution: {result}', file=sys.stderr) # manual execution my_worker() print(f'result at main thread after manual execution: {result}', file=sys.stderr) print(sys.version)  and the output is: variable address at main thread: 0x6ffffff39580 variable address at worker: 0x6ffffff39580 {'test': 'test'} result at main thread after execution: {} variable address at worker: 0x6ffffff39580 {'test': 'test'} result at main thread after manual execution: {'test': 'test'} 3.9.16 (main, Mar 8 2023, 22:47:22) [GCC 11.3.0]  My expectations were that result dictionary would show the changes done at the worker but it does not. What am I doing wrong?

Answer: You are using multiprocessing.Process. That is not a thread, that is a new process. It will inherit the whole address space of the parent process, but then work on its own copy. You will have to use threading.Thread if you want a thread. from threading import Thread def modify_dict(): result['test'] = 'test' print('Result in worker:', result) result = {} t = Thread(target=modify_dict) t.start() t.join() print('Result in main thread:', result)  Output: Result in worker: {'test': 'test'} Result in main thread: {'test': 'test'}
Question: I am trying to reach pages by hrefs nested to web elements. But the driver gives me two identical tabs when it click() on the href. It turns out that three tabs are open at the same time. Duplicate pages confuse further work with html. How can I get only one new tab by click? Python 3.11, PyCharm chrome_options = Options() chrome_options.add_argument("--headless") driver = webdriver.Chrome() driver.get("https://upn.ru/kupit/kvartiry") title = driver.title print(title) wait = WebDriverWait(driver, 5) original_window = driver.current_window_handle assert len(driver.window_handles) == 1 b_block = driver.find_element(By.CLASS_NAME, 'main-container-margins.width-100').click() wait.until(EC.number_of_windows_to_be(3)) for window_handle in driver.window_handles: if window_handle != original_window: driver.switch_to.window(window_handle) break title = driver.title print(title)  

Answer: The link is opened twice: • the normal <a> fires and Chrome opens a new tab • a JS listener on the same element calls window.open() and opens it again. WebDriver only replicates what the page tells the browser to do, so you have to avoid the click or neutralise the handler. from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC opt = webdriver.ChromeOptions() opt.add_argument('--headless') driver = webdriver.Chrome(options=opt) driver.get('https://upn.ru/kupit/kvartiry') w = WebDriverWait(driver, 5) link = w.until(EC.element_to_be_clickable( (By.CSS_SELECTOR, 'a.card-title'))) # pick the selector you need # simplest: open the URL yourself – no extra windows driver.get(link.get_attribute('href')) # --- or, if you really want to click --- # driver.execute_script( # "arguments[0].removeAttribute('target'); arguments[0].click();", link) # wait until EC.number_of_windows_to_be(2)  Only one new page is created, so further window-handle logic stays consistent.
Question: I often see code like this: try: some_operation() except Exception: logger.error("An error occurred while running the operation") raise Exception("A custom message")  Please ignore using the general Exception in this example, I know it's a bad practice. You can think of any other more specific subclasses of Exception. I don't feel fine with code like this. What is the purpose of catching an exception if it's raised again, maybe with modified message? If a developer does it for logging or changing the class of the exception is it OK and good approach? I understand try...except blocks (in any language) to really handle exceptions. The example I showed is for me not good, but maybe it's a common practice in python world? Is it? Is there a more proper way for logging in such cases?

Answer: This general pattern is an accepted practice. It allows a more meaningful message to be written for the user of the code as well as logging a more meaningful message for the developer. From the docs:  The most common pattern for handling Exception is to print or log the exception and then re-raise it (allowing a caller to handle the exception as well): import sys try: f = open('myfile.txt') s = f.readline() i = int(s.strip()) except OSError as err: print("OS error:", err) except ValueError: print("Could not convert data to an integer.") except Exception as err: print(f"Unexpected {err=}, {type(err)=}") raise
Question: I want to find the minimum value per row and create a new column indicating which of those columns has the lowest number. Unfortunately, it seems like pandas isn't immediately able to help in this regard. My research has led to the min() function, which does find the lowest for each row (when axis=1), but there's no further information beyond the number itself. initialDict = {"A":[6.53,11.47,92.08],"B":[9.11,8.15,12.49]} initialDf = pd.DataFrame.from_dict(initialDict,orient="index",columns=["Value 1","Value 2","Value 3"]) >>> initialDf Value 1 Value 2 Value 3 A 6.53 11.47 92.08 B 9.11 8.15 12.49 >>> initialDf.min(axis=1,numeric_only=True) A 6.53 # Value 1 - just a number is useless to me. B 8.15 # Value 2 - how do i access which columns these are?  My sample data is a lot larger than two rows, so ideally I'd want a vectorised solution. Can I somehow access which column has the lowest number per row and assign it to a new value?

Answer: A possible solution, which uses idxmin: initialDf.assign(min = initialDf.idxmin(axis=1))  Output:  Value 1 Value 2 Value 3 min A 6.53 11.47 92.08 Value 1 B 9.11 8.15 12.49 Value 2
Question: I'm building a chatbot to answer legal-related questions. I'm facing an issue with caching questions and responses — the goal is to retrieve an answer when someone asks a similar question that's already been saved. However, when I add new documents to the chatbot, the previously cached questions don't include information from these new files, so the responses become outdated and don't get updated accordingly. I've thought of two solutions:  When a cached question is asked, the system checks whether the number of information files has changed. If so, it fetches data from the newly added files. If there is relevant content, it generates a new answer that combines the new information with the previous response, then updates the cache.  When new files are added, a separate process is triggered to update the cached responses.   Both solutions raise concerns about the chatbot's performance. What approach would you recommend to keep the cache up-to-date without degrading performance?

Answer: Here is a very naive approach to such a scenario. Our assumptions are:  Your vector store initially contains N documents and any queries to which relevant context exists is cached.  When the vector store is updated (i.e count of documents change) you want the cached items to be updated.   from langchain_core.caches import BaseCache, RETURN_VAL_TYPE from typing import Any, Dict, Optional class MetadataAwareCache(BaseCache): def __init__(self, doc_count: int): super().__init__() self._cache = {} self._doc_count = doc_count # Initially set a document count for reference def update_document_count(self, doc_count: int): if self._doc_count == doc_count: return self._doc_count = doc_count for args, _old_response in self._cache.items(): prompt, llm_string = args[0], args[1] response, metadata = self.regenerate_cache(prompt, llm_string) self.update(prompt, llm_string, response, metadata) def regenerate_cache(self, prompt: str, llm_input: str): # Regenerate response with new information response = "New LLM Response" metadata = {} return response, metadata # Cache lookup def lookup(self, prompt: str, llm_string: str) -> Optional[RETURN_VAL_TYPE]: cache_entry = self._cache.get((prompt, llm_string)) if cache_entry: return cache_entry["value"] return None # Update cache def update( self, prompt: str, llm_string: str, value: RETURN_VAL_TYPE, metadata: Dict[str, Any] = None, ) -> None: self._cache[(prompt, llm_string)] = { "value": value, "metadata": metadata or {}, }  To use the cache from langchain.globals import set_llm_cache cache = MetadataAwareCache() set_llm_cache(cache)  Considerations:  If you are worried about performance and potentially the cost of re-generating the cache you can consider the frequency at which new information is likely to be added. If it is far and between it could be better to expire cache often rather than pay the cost to re-create cache that might not be re-used.  If the updates are more often, you could make it a scheduled process.  Alternatively you can add more metadata for each document and re-create only those cache that fall under such category.