eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have the following line that configures my logger instance logger.add(sys.stderr, level=log_level, format="<green>{time:YYYY-MM-DD HH:mm:ss.SSS zz}</green> | <level>{level: <8}</level> | <yellow>Line {line: >4} ({file}):</yellow> <b>{message}</b>", colorize=True, backtrace=True, diagnose=True)  I want to configure my logger to use the above, and also output to a file, so that when I call logger.whatever() it outputs to the terminal and to a log file This way, if I'm doing dev work and running the file directly, I can see the output in my terminal, and when the code is being run on a server by a cronjob, it can log to a file I don't understand the whole concept of the "sinks" so sorry if this is an easy question

Answer: In the context of logging, a "sink" refers to an output destination for log records. It can be a stream (such as sys.stderr or sys.stdout), a file path, or any custom function that accepts the logged message an input. See documentation for more details. Using Loguru, you can add as many sinks as you like. This means that in your case, you can have two sinks: one writing to sys.stderr and another one writing to file.log. log_level = "DEBUG" log_format = "<green>{time:YYYY-MM-DD HH:mm:ss.SSS zz}</green> | <level>{level: <8}</level> | <yellow>Line {line: >4} ({file}):</yellow> <b>{message}</b>" logger.add(sys.stderr, level=log_level, format=log_format, colorize=True, backtrace=True, diagnose=True) logger.add("file.log", level=log_level, format=log_format, colorize=False, backtrace=True, diagnose=True)  Each time a message is logged with logger.info() for example, it will be sent to each of the added sinks and processed based on the the corresponding configuration.
Question: I have an A/B test split based on the result of Java hashCode() function applied to user's id (a string). I want to emulate that split in my dataframe to analyse the results. Is there a python equivalent for that function? Or maybe a documentation on the specific hashing algorithm inside hashCode() so I can produce that function myself? Thanks I searched for the documentation but couldn't find the specific details

Answer: According to java String source code, the hash implementation is: public int hashCode() { if (cachedHashCode != 0) return cachedHashCode; // Compute the hash code using a local variable to be reentrant. int hashCode = 0; int limit = count + offset; for (int i = offset; i < limit; i++) hashCode = hashCode * 31 + value[i]; return cachedHashCode = hashCode; }  You can transfer this to Python (w/o caching): class JavaHashStr(str): def __hash__(self): hashCode = 0 for char in self: hashCode = hashCode * 31 + ord(char) return hashCode >>> j = JavaHashStr("abcd") >>> hash(j) 2987074 # same as java >>> j = JavaHashStr("abcdef") >>> hash(j) 2870581347 # java: -1424385949  Note, Python ints do not overflow like java, so this is wrong for many cases. You would have to add a simulation for the overflow (Update: thx to @PresidentJamesK.Polk for the improved version, SO thread on the topic): class JavaHashStr(str): def __hash__(self): hashCode = 0 for char in self: hashCode = (hashCode * 31 + ord(char)) & (2**32 - 1) # unsigned if hashCode & 2**31: hashCode -= 2**32 # make it signed return hashCode  Now, even overflowing hashes behave the same: >>> j = JavaHashStr("abc") >>> hash(j) 96354 >>> j = JavaHashStr("abcdef") >>> hash(j) -1424385949 # Java hash for "abcdef"  This might still be off for characters from the latter unicode panes like emojis or the like. But for the most common punctuation and latin-based characters, this should work.
Question: I have a dictionary like the following: testProps = {"Key1":[], "Key2":False, "Key3":True, "Key4":[], "Key5":False}  I want to convert it to a Pandas DataFrame with newTestProps = pd.DataFrame.from_dict(testProps,orient="index")  but it is throwing me the following error. TypeError: object of type 'bool' has no len() When it should be creating a DataFrame like the following:  Which is bizarre because I've had these two lines of code in my program for a long time and it hasn't thrown this error until now. Is it possible that there's a bug with Pandas?

Answer: To create the dataframe you can use next example: testProps = {"Key1": [], "Key2": False, "Key3": True, "Key4": [], "Key5": False} df = pd.DataFrame({'Index': testProps.keys(), 0: testProps.values()}) print(df)  Prints:  Index 0 0 Key1 [] 1 Key2 False 2 Key3 True 3 Key4 [] 4 Key5 False
Question: I am using undetected_chromedriver, but I am getting the error This version of ChromeDriver only supports Chrome version 114 Current browser version is 103.0.5060.53  My code: import undetected_chromedriver as uc uc.TARGET_VERSION = 103 options = uc.ChromeOptions() options.add_argument('--no-sandbox') options.add_argument('--disable-dev-shm-usage') driver = uc.Chrome(options=options, version_main=103, patcher_force_close=True) driver.get('https://nowsecure.nl') driver.quit() 

Answer: You can use SeleniumBase's UC Mode to use undetected-chromedriver with any browser version (it automatically downloads the correct driver if missing). First pip install seleniumbase, and then run the following script with python: from seleniumbase import Driver import time driver = Driver(uc=True, incognito=True) driver.get("https://nowsecure.nl/#relax") time.sleep(8) driver.quit()
Question: I want to print "getting x attr" whenever I use the dot notation to get an attribute. For example class MyClass: my_attribute = "abc" print(MyClass.my_attribute)  I'd expect it to output like this: >>> getting my_attribute attr >>> abc  I'm not instantiating the class. I tried to add a print statement to MyClass.__getattr__() and MyClass.__getattribute__() but none of these will display anything. I tried the following: class MyClass: def __getattr__(self, key): print("running __getattr__") return super().__getattr__(key) def __getattribute__(self, __name: str) -> Any: print("running __getattribute__") return super().__getattribute__(__name) my_attribute = "abc" 

Answer: MyClass (the class object) is itself an object. You want to define your custom __getattribute__ method on the class that MyClass is an instance of. To do that, use a metaclass: class MyMeta(type): def __getattribute__(self, item): print(f"getting {item}") return super().__getattribute__(item) class MyClass(metaclass=MyMeta): my_class_attribute = "abc"  then print(type(MyClass)) print(MyClass.my_class_attribute)  outputs <class '__main__.MyMeta'> getting my_class_attribute abc
Question: I tried to annotate a line plot with ax.annotate as follows. import numpy as np import matplotlib.pyplot as plt x_start = 0 x_end = 200 y_start = 20 y_end = 20 fig, ax = plt.subplots(figsize=(5,5),dpi=600) ax.plot(np.asarray([i for i in range(0,1000)])) ax.annotate('', xy=(x_start, y_start), xytext=(x_end, y_end), xycoords='data', textcoords='data', arrowprops={'arrowstyle': '|-|'}) plt.show()  which gave a plot (zoomed in)  Although I have specified x_start to be 0 and x_end to be 200, the actual start is greater than 0 and actual end is smaller than 200 on the x-axis. How do I correctly line up this annotation with the set coordinates?

Answer: By default, the arrow is shrunk by 2 points on both ends (see the doc). You can set shrinkA and shrinkB to 0 to align with your x-axis: import numpy as np import matplotlib.pyplot as plt x_start = 0 x_end = 200 y_start = 20 y_end = 20 fig, ax = plt.subplots(figsize=(5,5),dpi=600) ax.plot(np.asarray([i for i in range(0,1000)])) ax.annotate('', xy=(x_start, y_start), xytext=(x_end, y_end), xycoords='data', textcoords='data', arrowprops={'arrowstyle': '|-|', 'shrinkA': 0, 'shrinkB': 0}) plt.show()  Output:
Question: I want to create a Pydantic custom field. The main goal of this validator is to be able to accept two data types: "str" and "None". If the value is "None", it should return an empty string. I tried to do it as follows: from pydantic import BaseModel class EmptyStringField: @classmethod def __get_validators__(cls): yield cls.validate @classmethod def validate(cls, v): if v is None: return "" return str(v) class Model(BaseModel): url: EmptyStringField model = Model(url=None) print(model.url)  However, I'm getting the following error: url none is not an allowed value (type=type_error.none.not_allowed) 

Answer: If all you want is for the url field to accept None as a special case, but save an empty string instead, you should still declare it as a regular str type field. You can handle the special case in a custom pre=True validator. No need for a custom data type there. from pydantic import BaseModel, validator class Model(BaseModel): url: str @validator("url", pre=True) def none_to_empty(cls, v: object) -> object: if v is None: return "" return v model = Model(url=None) print(model.json()) # {"url": ""}   Update If you don't want to repeat the validator in different models for different fields, you can define a catch-all pre=True validator on a custom base model and implement some sort of logic to discern, which fields on the model to process and how. One option is to utilize typing.Annotated to "package" any given type with some custom conversion function that should be called. The catch-all validator would then check every field for Annotated metadata and if it finds a function, it would call that on the value. Here is a working example: from typing import Annotated, get_origin from pydantic import BaseModel as PydanticBaseModel, validator from pydantic.fields import ModelField class BaseModel(PydanticBaseModel): @validator("*", pre=True) def process_annotated(cls, v: object, field: ModelField) -> object: if get_origin(field.annotation) is not Annotated: return v func = field.annotation.__metadata__[0] if not callable(func): return v return func(v) StrOrNone = Annotated[str, lambda v: "" if v is None else v] class Model1(BaseModel): url: StrOrNone print(Model1(url=None).json()) # {"url": ""}
Question: I want to extract the first 3 elements from a list of strings from the '1/1' column. My df_unique looks like that:  1/1 0/0 count 0 ['P1-12', 'P1-22', 'P1-25', 'P1-26', 'P1-28', 'P1-6', 'P1-88', 'P1-93'] ['P1-89', 'P1-90', 'P1-92', 'P1-95'] 1 1 ['P1-12', 'P1-22', 'P1-25', 'P1-26', 'P1-6', 'P1-89', 'P1-92', 'P1-95'] ['P1-28', 'P1-90', 'P1-93'] 1 2 ['P1-12', 'P1-22', 'P1-25', 'P1-26', 'P1-88', 'P1-89', 'P1-92', 'P1-93', 'P1-95'] ['P1-28', 'P1-6', 'P1-90'] 1 3 ['P1-12', 'P1-22', 'P1-25', 'P1-26', 'P1-88', 'P1-89', 'P1-92', 'P1-93'] ['P1-28', 'P1-6', 'P1-90'] 1  I've tried to use different solutions: df_extract_3 = df_unique['1/1'].str.split().map(lambda lst: [string[0:3] for string in lst])  but the result looks like that: 0 [['P, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1] 1 [['P, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1] 2 [['P, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1] 3 [['P, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1, 'P1]  And the second solution: df_extract_3 = df_unique['1/1'].str[0:3]  gives: 0 ['P 1 ['P 2 ['P 3 ['P  When I try to add split : df_extract_3 = df_unique['1/1'].str.split().str[0:3]  the final result is: 0 [['P1-12',, 'P1-22',, 'P1-25',] 1 [['P1-12',, 'P1-22',, 'P1-25',] 2 [['P1-12',, 'P1-22',, 'P1-25',] 3 [['P1-12',, 'P1-22',, 'P1-25',]  What should I change to receive 'normal' output like: 0 ['P1-12', 'P1-22', 'P1-25'] 1 ['P1-12', 'P1-22', 'P1-25'] 2 ['P1-12', 'P1-22', 'P1-25'] 3 ['P1-12', 'P1-22', 'P1-25']  ? I know it can be easy modification but I've stuck and messed with that... Thanks a lot!

Answer: First convert your strings to read lists, then slice with str: import ast df_unique['1/1'] = df_unique['1/1'].apply(ast.literal_eval) df_unique['0/0'] = df_unique['0/0'].apply(ast.literal_eval) df_extract_3 = df_unique['1/1'].str[:3] print(df_extract_3)  Or in one shot: df_extract_3 = df_unique['1/1'].apply(lambda x: ast.literal_eval(x)[:3])  Output: 0 [P1-12, P1-22, P1-25] 1 [P1-12, P1-22, P1-25] 2 [P1-12, P1-22, P1-25] 3 [P1-12, P1-22, P1-25] Name: 1/1, dtype: object
Question: Having dataframe import pandas as pd df = pd.DataFrame( { "group0": [1, 1, 2, 2, 3, 3], "group1": ["1", "1", "1", "2", "2", "2"], "relevant": [True, False, False, True, True, True], "value": [0, 1, 2, 3, 4, 5], } )  I wish to produce a target target = pd.DataFrame( { "group0": [1, 2, 2, 3], "group1": ["1","1", "2", "2",], "value": [0, 2, 3, 5], } )  where "value" has been chosen by  Maximum of all positive "relevant" indices in "value" column Otherwise maximum of "value" if no positive "relevant" indices exist  This would be produced by def fun(x): tmp = x["value"][x["relevant"]] if len(tmp): return tmp.max() return x["value"].max()  were x a groupby dataframe. Is it possible to achive the desired groupby reduction efficiently? EDIT: with payload from time import perf_counter() df = pd.DataFrame( { "group0": np.random.randint(0, 30,size=10_000_000), "group1": np.random.randint(0, 30,size=10_000_000), "relevant": np.random.randint(0, 1, size=10_000_000).astype(bool), "value": np.random.random_sample(size=10_000_000) * 1000, } ) start = perf_counter() out = (df .sort_values(by=['relevant', 'value']) .groupby(['group0', 'group1'], as_index=False) ['value'].last() ) end = perf_counter() print("Sort values", end - start) def fun(x): tmp = x["value"][x["relevant"]] if len(tmp): return tmp.max() return x["value"].max() start = perf_counter() out = df.groupby(["group0", "group1"]).apply(fun) end = perf_counter() print("Apply", end - start) #Sort values 14.823943354000221 #Apply 1.5050544870009617  .apply-solution got time of 1.5s. The solution with sort_values performed with 14.82s. However, reducing sizes of the test groups with ... "group0": np.random.randint(0, 500_000,size=10_000_000), "group1": np.random.randint(0, 100_000,size=10_000_000), ...  led to vastly superior performance by the sort_values solution. (15.29s versus 1423.84s). sort_values solution by @mozway is preferred, unless user specifically knows that data contains small group counts.

Answer: Sort the values to put True, then highest number last and use a groupby.last: out = (df .sort_values(by=['relevant', 'value']) .groupby(['group0', 'group1'], as_index=False) ['value'].last() )  Output:  group0 group1 value 0 1 1 0 1 2 1 2 2 2 2 3 3 3 2 5  Intermediate before aggregation: * selected rows  group0 group1 relevant value 1 1 1 False 1 2 2 1 False 2 * 0 1 1 True 0 * 3 2 2 True 3 * 4 3 2 True 4 5 3 2 True 5 *
Question: Using Pillow 9.5.0, Python 3.11.4, Tk 8.6, and Windows 11 (version 22H2) On the program i'm working on, I have a tkinter canvas that saves a postscript file, which is then imported into a PIL Image and shown. The problem is that it always has a white background, even when I changed the canvas background color Here's an excerpt of my example self.canvas.update() self.canvas.postscript(file='temp_{insert_garbage_here}.eps') img = Image.open('temp_{insert_garbage_here}.eps') img.show()  Is there any way to fix this, so that the background is transparent? Btw, I tried using mask = Image.new('L', img.size, color=255) img.putalpha(mask)  but that didn't change anything And this is the file the postscript exported: https://www.dropbox.com/scl/fi/fz0xt6is5aicgnqvkcskd/temp_-insert_garbage_here.eps?dl=0&rlkey=yd2pjb61t1rmydm9ky704zcfl

Answer: After opening the image, you can call the load() method with transparency=True to tell ghostscript to load the EPS (Encapsulated PostScript) image with a transparent background: from PIL import Image im = Image.open('a.eps') im.load(transparency=True) # Check we now have RGBA inage print(im.mode) # prints "RGBA"
Question: I am following this tutorial to see why Python doesn't recognize an environment variable that was set at the Conda prompt (using CMD syntax, as this is an Anaconda installation on Windows 10). It requires the os module, and as part of my getting familiar with Python, I decided to test whether os was already imported. Testing the presence of a module requires the sys module (as described here). Strangely, right after importing sys, I found that os was imported without me having to do so. I find this odd, as most of my googling shows that you have to import them individually, e.g., here. Does importing sys also impart os, as it seems to? If so, why is it common to import both individually? I can't test for the presence of os before importing sys, as I need sys to test for the presence of modules. Here is the code that shows the apparent presence of os from importing sys, formatted for readability. It starts from the Conda prompt in Windows. The Conda environment is "py39", which is for Python 3.9: (py39) C:\Users\User.Name > python Python 3.9.16 (main, Mar 8 2023, 10:39:24) [MSC v.1916 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> "os" in sys.modules True  Afternote: Thanks to Zero's answer, I found this code to be more what I'm looking for. After loading sys, the appropriate test is ( 'os' in sys.modules ) and ( 'os' in dir() ): (py39) C:\Users\User.Name > python 'os' in dir() # False import sys 'os' in sys.modules , 'os' in dir() # (True, False) ( 'os' in sys.modules ) and ( 'os' in dir() ) # False import os 'os' in sys.modules , 'os' in dir() # (True, True) ( 'os' in sys.modules ) and ( 'os' in dir() ) # True  sys.modules shows whether the module has been imported anywhere (presumably in the code that the Python interpreter has executed) while dir() indicates whether the module name is in the current namespace. Thanks to Carcigenicate for clarifying this point, and I hope that I understood it properly.

Answer: sys uses the os module, but doesn't import it to your code. You can confirm it through the code. In [1]: import sys In [2]: os.getcwd() --------------------------------------------------------------------------- NameError Traceback (most recent call last) Cell In [2], line 1 ----> 1 os.getcwd() NameError: name 'os' is not defined
Question: I have a DataFrame with the following columns: INVOICE_DATE, COUNTRY, CUSTOMER_ID, INVOICE_ID, DESCRIPTION, USIM, and DEMANDQTY. I want to filter the DataFrame based on specific conditions.  The condition is that if the DESCRIPTION column contains the words "kids" or "baby", I want to include all the values from that INVOICE_ID in the filtered DataFrame. In other words, at least one item in the transaction should belong to the kids or baby category for the entire transaction to be included. I tried using the str.contains() method in combination with a regular expression pattern, but I'm having trouble getting the desired results. Here's my code: import pandas as pd # Assuming the DataFrame is named 'df' # Filter the DataFrame based on the condition filtered_df = df[df['DESCRIPTION'].str.contains('kids|baby', case=False, regex=True)] # Print the filtered DataFrame filtered_df  However, this code does not provide the expected results. It filters the data frame based on individual rows rather than considering the entire transaction. Please find below the test data: - import pandas as pd import random import string import numpy as np random.seed(42) np.random.seed(42) num_transactions = 100 max_items_per_transaction = 6 # Generate a list of possible items possible_items = [ "Kids T-shirt", "Baby Onesie", "Kids Socks", "Men's Shirt", "Women's Dress", "Kids Pants", "Baby Hat", "Women's Shoes", "Men's Pants", "Kids Jacket", "Baby Bib", "Men's Hat", "Women's Skirt", "Kids Shoes", "Baby Romper", "Men's Sweater", "Kids Gloves", "Baby Blanket" ] # Create the DataFrame rows = [] for i in range(num_transactions): num_items = random.randint(1, max_items_per_transaction) items = random.sample(possible_items, num_items) invoice_dates = pd.date_range(start='2022-01-01', periods=num_items, freq='D') countries = random.choices(['USA', 'Canada', 'UK'], k=num_items) customer_id = i + 1 invoice_id = 1001 + i for j in range(num_items): item = items[j] usim = ''.join(random.choices(string.ascii_uppercase + string.digits, k=6)) # Generate a random 6-character USIM value demand_qty = random.randint(1, 10) row = { 'INVOICE_DATE': invoice_dates[j], 'COUNTRY': countries[j], 'CUSTOMER_ID': customer_id, 'INVOICE_ID': invoice_id, 'DESCRIPTION': item, 'USIM': usim, 'DEMANDQTY': demand_qty } rows.append(row) df = pd.DataFrame(rows) # Print the DataFrame df  Can anyone please guide me on how to properly filter the DataFrame based on the described condition? I would greatly appreciate any help or suggestions. Thank you!

Answer: Suppose the following dataframe: >>> df DESCRIPTION INVOICE_ID 0 kids 123 1 hello 123 2 world 123 3 another 456 4 one 456  You can want to keep INVOICE_ID=123 because 'kids' is in the description of row 0: m = df['DESCRIPTION'].str.contains('kids|baby', case=False, regex=True) filtered_df = df[m.groupby(df['INVOICE_ID']).transform('max')]  Output: >>> filtered_df DESCRIPTION INVOICE_ID 0 kids 123 1 hello 123 2 world 123
Question: I have a dataframe with a column of numbers that look like this: data = [291.79, 499.31, 810.93, 1164.25] df = pd.DataFrame(data, columns=['Onset'])  Is there an elegant way to round down the odd numbers to the nearest even number? The even numbers should be rounded to the nearest even integer. This should be the output: data = [290, 498, 810, 1164] df = pd.DataFrame(data, columns=['Onset'])  Thank you!

Answer: I'd use modulo operator: df['Onset_new'] = (i:=df['Onset'].astype(int)) - i % 2 print(df)  Prints:  Onset Onset_new 0 291.79 290 1 499.31 498 2 810.93 810 3 1164.25 1164
Question: I have 2 models, Item and Category, the model Item has category field as a foreign key In my views.py I get a list of queries from a POST request queries = ['category1', 'category2', 'category3', ...] I don't know the number of the queries I will get from the request, and I want to filter the Item model based on category field I tried this from django.db.models import Q from .models import Item, Category from django import views class myView(views.View): def post(self, request): queries = request.POST.get('queries', '') if queries: queriesList = [] queries = queries.split(',') # queries = ['category1', 'category2', ....] for query in queries: queriesList.append(Q(category__icontains=query)) queryset = Item.objects.filter(*queriesList) # this will do AND but won't do OR # I tried: queryset = Item.objects.filter([q | Q() for q in queriesList]) but it didn't work  Also I tried queryset = Item.objects.filter(category__in=queries) but it's case sensitive

Answer: You can use Q.OR as _connector: from django.db.models import Q from django import views from .models import Category, Item class myView(views.View): def post(self, request): queries = request.POST.get('queries', '') if queries: queryset = Item.objects.filter( *[Q(category__icontains=query) for query in queries.split(',')], _connector=Q.OR ) searching is however normally done through a GET request, such that the search query appears in the URL, and thus the URL can be copied, bookmarked, etc. with the search query included.
Question: I have a DataFrame with an index column that uses pandas TimeStamps. How can I get the values of the index as if it were a normal column (df["index"])?

Answer: You can get the index values from your dataframe by the following: df.index.values  The above will return an array of the index values of your dataframe. You can simply store that in a variable. You can also get the values as a list by the following: list(df.index.values)