eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I would like to binarize a png image. I would like to use Pillow if possible. I've seen two methods used: image_file = Image.open("convert_image.png") # open colour image image_file = image_file.convert('1') # convert image to black and white  This method appears to handle a region filled with a light colour by dithering the image. I don't want this behaviour. If there is, for example, a light yellow circle, I want that to become a black circle. More generally, if a pixel's RGB is (x,y,z) the I want the pixel to become black if x<=t OR y<=t OR z<=t for some threshold 0<t<255 I can covert the image to greyscale or RGB and then manually apply a threshold test but this seems inefficient. The second method I've seen is this: threshold = 100 im = im2.point(lambda p: p > threshold and 255)  from here I don't know how this works though or what the threshold is or does here and what "and 255" does. I am looking for either an explanation of how to apply method 2 or an alternative method using Pillow.

Answer: I think you need to convert to grayscale, apply the threshold, then convert to monochrome. image_file = Image.open("convert_iamge.png") # Grayscale image_file = image_file.convert('L') # Threshold image_file = image_file.point( lambda p: 255 if p > threshold else 0 ) # To mono image_file = image_file.convert('1')  The expression "p > threshhold and 255" is a Python trick. The definition of "a and b" is "a if a is false, otherwise b". So that will produce either "False" or "255" for each pixel, and the "False" will be evaluated as 0. My if/else does the same thing in what might be a more readable way.
Question: I want to read a text file and check for strings with open(my_file,'r') as f: for line in f: if 'text1' in line: f.next() if 'text2' in line: # do some processing  I want to first find the text 'text1' at the beginning of the line then if found I want to check the next line for 'text2' if found then I will do some other processing. Appears that the f.next() is not moving to the next line.

Answer: The variable line does not magically get updated when you call f.next() (or next(f) in Python 3). You would instead have to assign the line returned by next to a variable and test against it: with open(my_file,'r') as f: for line in f: if 'text1' in line: try: next_line = next(f) except StopIteration: break # in case we are already at the end of file if 'text2' in next_line: # do some processing with line and next_line  If there can be cases where there are two consecutive lines with text1 followed by a line of text2, however, calling next(f) would make it consume the second line of text1, making it unable to match text1 in the next iteration. To remedy that, you can use the pairwise recipe in the itertools documentation to iterate 2 lines at a time over the file instead: import itertools def pairwise(iterable): a, b = itertools.tee(iterable) next(b, None) return zip(a, b) with open(my_file,'r') as f: for line, next_line in pairwise(f): if 'text1' in line and 'text2' in next_line: # do some processing with line and next_line
Question: I'm trying to use PySpark to read in a CSV file with many columns. The inferschema option is great at inferring majority of the columns' data types. If I want to override just one of the columns types that were inferred incorrectly, what is the best way to do this? I have this code working, but it makes PySpark import only the one column that is specified in the schema, which is not want I want. schema = StructType() \ .add("column_one_of_many", StringType(), True) spark.read.format('com.databricks.spark.csv') \ .option('delimited',',') \ .option('header','true') \ .option('inferschema', 'true') \ .schema(self.schema) \ .load('dbfs:/FileStore/some.csv')  Is what I'm asking for even possible? Thank you for your time and guidance :)

Answer: Easier way would be using .withColumn and casting column_one_of_many  as string. Example from pyspark.sql.types import * spark.read.format('com.databricks.spark.csv') \ .option('delimited',',') \ .option('header','true') \ .option('inferschema', 'true') \ .load('dbfs:/FileStore/some.csv')\ .withColumn("column_one_of_many",col("column_one_of_many").cast("string"))   Other way would be defining all the columns in schema then exclude the inferschema just use .schema option to read the csv file.
Question: I have a list of items that should be inserted in a list-like data structure one after the other, and I have the indexes at which each item should be inserted. For example: items = ['itemX', 'itemY', 'itemZ'] indexes = [0, 0, 1]  The expected result is to have a list like this: result = ['itemY', 'itemZ', 'itemX']. I'm able to get this result with this simple approach: result = [] for index, item in zip(indexes, items): result.insert(index, item)  However, this is a very slow approach once lists become huge (the complexity is O(n^2)). Is there any (relatively simple to implement) way to improve my basic approach? I guess I have to look at other data structures while I insert elements and finally transform that data structure into my result list. Are trees a good option? Insert could be done maybe in O(log(n)) (instead of O(n)), but which specific tree-like structure should I use? Or maybe something good can be achieved by just looking at all the indexes together (instead of using them one by one). This is probably the worst case for my slow approach (always insert items at the beginning of the list): n = 10**6 # some large number items = list(range(n)) indexes = [0] * n 

Answer: Here's python code for a treap with a size decoration that allows insertion at specific indexes, and reordering of whole contiguous sections. It was adapted from C++ code, Kimiyuki Onaka's solution to the Hackerrank problem, "Give Me the Order." (I cannot guarantee that this adaptation is bug free -- a copy of the original code is available in the description of this question.) import random class Treap: def __init__(self, value=None): self.value = value self.key = random.random() self.size = 1 self.left = None self.right = None def size(t): return t.size if t else 0 def update(t): if t: t.size = 1 + size(t.left) + size(t.right) return t def merge(a, b): if not a: return b if not b: return a if a.key > b.key: a.right = merge(a.right, b) return update(a) else: b.left = merge(a, b.left) return update(b) def split(t, i): if not t: return None, None if i <= size(t.left): u, t.left = split(t.left, i) return u, update(t) else: t.right, u = split(t.right, i - size(t.left) - 1) return update(t), u def insert(t, i, value): left, right = split(t, i) u = Treap(value) return merge(merge(left, u), right) def inorder(treap): if not treap: return if treap.left: inorder(treap.left) print(treap.value) if treap.right: inorder(treap.right)  Output: lst = ['itemX', 'itemY', 'itemZ'] idxs = [0, 0, 1] t = None for i in range(len(lst)): t = insert(t, idxs[i], lst[i]) inorder(t) """ itemY itemZ itemX """
Question: How to explode pandas data frame? Input df:  Required output df: +----------------+------+-----+------+ |level_2 | date | val | num | +----------------+------+-----+------+ | name_1a | 2020 | 1 | null | | name_1b | 2019 | 2 | null | | name_1b | 2020 | 3 | null | | name_10000_xyz | 2018 | 4 | str | | name_10000_xyz | 2019 | 5 | null | | name_10000_xyz | 2020 | 6 | str | +------------------------------------+  To reproduce input df: import pandas as pd pd.set_option('display.max_colwidth', None) data={'level_2':{1:'name_1a',3:'name_1b',5:'name_10000_xyz'},'value':{1:[{'date':'2020','val':1}],3:[{'date':'2019','val':2},{'date':'2020','val':3}],5:[{'date':'2018','val':4,'num':'str'},{'date':'2019','val':5},{'date':'2020','val':6,'num':'str'}]}} df = pd.DataFrame(data) 

Answer: Explode the dataframe on value column, then pop the value column and create a new dataframe from it then join the new frame with the exploded frame. s = df.explode('value', ignore_index=True) s.join(pd.DataFrame([*s.pop('value')], index=s.index))    level_2 date val num 0 name_1a 2020 1 NaN 1 name_1b 2019 2 NaN 2 name_1b 2020 3 NaN 3 name_10000_xyz 2018 4 str 4 name_10000_xyz 2019 5 NaN 5 name_10000_xyz 2020 6 str
Question: Applying mask with NumPy or OpenCV is a relatively straightforward process. However, if I need to use masked image in loss calculations of my optimization algorithm, I need to employ exclusively PyTorch, as doing otherwise interferes with gradient computations. Assuming that I have an image tensor [1, 512, 512, 3] (batch, height, width, channels) and a mask tensor [1, 20, 512, 512] (batch, channels, height, width) where every channel corresponds to one of 20 segmentation classes, I want to get a masked image tensor that fills every pixel with black (0, 0, 0), except for those belonging to one or more specified segmentation classes. Here is how it can be done with numpy: import numpy as np import torch # Create dummy image and mask image_tensor = torch.randn([1, 512, 512, 3]) mask_tensor = torch.randn([1, 20, 512, 512]) # Apply argmax to mask mask_tensor = torch.max(mask_tensor, 1)[1] # -> 1, 512, 512 # Define mask function def selective_mask(image_src, mask, dims=[]): h, w = mask.shape background = np.zeros([h, w, 3], dtype=np.uint8) for j_, j in enumerate(mask[:, :]): for k_, k in enumerate(j): if k in dims: background[j_, k_] = image_src[j_, k_] output = background return output # Convert tensors to numpy: image = image_tensor.squeeze(0).cpu().numpy() mask = mask_tensor.squeeze(0).cpu().nmpy() # Apply mask function for several classes image_masked = selective_mask(image, mask, dims=[5, 6, 8])  How should my code be changed to bring it in line with the PyTorch requirements?

Answer: First of all, the definition of the function selective_mask is far for what You may call 'straightforward'. The key point in using numpy (and torch, which is designed to be mostly compatible) is to take advantage of the vectorization of operations and to avoid using loops, which are not parallelizable. If You rewrite the said function in this manner: def selective_mask(image_src, mask, channels=[]): mask = mask[np.array(channels).astype(int)] return np.sign(np.sum(mask, axis=0), dtype=image_src.dtype) * image_src  it will turn out that You can actually do the same with pytorch tensors (here no need to squeeze the batch (first) dimension): def selective_mask_t(image_src, mask, channels=[]): mask = mask[:, torch.tensor(channels).long()] mask = torch.sgn(torch.sum(mask, dim=1)).to(dtype=image_src.dtype).unsqueeze(-1) return mask * image_src  Also, You probably want to produce the mask itself this way: (BTW using a combination of max and sgn here should actually work faster than setting elements indexed by argmax) # Create dummy image and mask image_tensor = torch.randn([1, 512, 512, 3]) mask_tensor = torch.randn([1, 20, 512, 512]) # Discreticize the mask (set to one in the channel with the highest value) -> 1, 20, 512, 512 mask_tensor = torch.sgn(mask_tensor - torch.max(mask_tensor, 1)[0].unsqueeze(1)) + 1.  Then it should work just fine: print(selective_mask_t(image_tensor, mask_tensor, [5, 6, 8]))
Question: I have a pandas frame where the index is a DateTimeIndex and I am trying to infer its frequency and it is coming up as None. df.index DatetimeIndex(['2020-08-24 00:00:00', '2020-08-24 00:01:00', '2020-08-24 00:02:00', '2020-08-24 00:03:00', '2020-08-24 00:04:00', '2020-08-24 00:05:00', '2020-08-24 00:06:00', '2020-08-24 00:07:00', '2020-08-24 00:08:00', '2020-08-24 00:09:00', ... '2021-08-22 23:51:00', '2021-08-22 23:52:00', '2021-08-22 23:53:00', '2021-08-22 23:54:00', '2021-08-22 23:55:00', '2021-08-22 23:56:00', '2021-08-22 23:57:00', '2021-08-22 23:58:00', '2021-08-22 23:59:00', '2021-08-23 00:00:00'], dtype='datetime64[ns]', name='timestamp', length=307668, freq=None  As you can see the data is coming at a 1 minute frequency but when I do something like: pd.infer_freq(df.index) # returns None  Is there some other way to figure out the input frequency? Is there a method that would be robust to some missing data?

Answer: freq is already None in this case, so you should try: >>> pd.to_timedelta(np.diff(df.index).min()) Timedelta('0 days 00:01:00') >>>  Or just: >>> np.diff(df.index).min() numpy.timedelta64(60000000000,'ns')
Question: I'm using Pandas 1.3.2 in a Conda environment. When importing pandas on a Jupyter Notebook: import pandas as pd  I get the error: ImportError: cannot import name 'DtypeArg' from 'pandas._typing' (C:\Users\tone_\anaconda3\envs\spyder\lib\site-packages\pandas\_typing.py)  I've seen similar questions, but so far no solution. Can anyone help?

Answer: According to the answer provided in this post it is a bug in pandas==1.3.1. A possible solution is to downgrade it to some earlier version, e.g pip install pandas==1.3.0
Question: I have a pandas dataframe where the column names are frequencies in 1 Hz steps, each row is a participant id, and the values are an amplitude^2 value for the participant in each respective frequency. I am trying to plot a time-series of the data where the x axis are the frequencies, and the y axis is the amplitude^2 value, in "spaghetti plot" style, i.e. there is one line plotted for each row of my dataframe: Here is a small snippet of my data: data = [['1', 9.45e-09, 9.85e-09, 8.33e-09, 6.06e-09, 4.80e-09, 4.08e-09], ['2', 1.30e-08, 1.25e-08, 8.99e-09, 6.25e-09, 4.44e-09, 3.45e-09], ['3', 9.32e-09, 8.60e-09, 5.67e-09, 3.68e-09, 2.53e-09, 1.75e-09]] fft_df = df = pd.DataFrame(data, columns = ['id', '1','2','3','4','5','6']).set_index('id') # display(fft_df) 1 2 3 4 5 6 id 1 9.450000e-09 9.850000e-09 8.330000e-09 6.060000e-09 4.800000e-09 4.080000e-09 2 1.300000e-08 1.250000e-08 8.990000e-09 6.250000e-09 4.440000e-09 3.450000e-09 3 9.320000e-09 8.600000e-09 5.670000e-09 3.680000e-09 2.530000e-09 1.750000e-09  Using matplotlib, if I use the fft_df column names as the x argument, and the fft_df column mean as the y argument, matplotlib will return a lineplot. However if I remove the .mean() from the y input it will return an error. I cannot seem to figure out how to plot one line for each row in fft_df: plt.figure(figsize=(10, 10)) plt.ylabel('Absolute Power (log)',fontsize=12) plt.xlabel('Frequencies',fontsize=12) plt.plot(fft_df.columns,fft_df.mean())  

Answer: I think, the easiest solution would be to transpose your DataFrame and then use pandas' plotting method. This is somewhat based on this answer. The code would look like this: import pandas as pd import matplotlib.pyplot as plt data = [['1', 9.45e-09, 9.85e-09, 8.33e-09, 6.06e-09, 4.80e-09, 4.08e-09], ['2', 1.30e-08, 1.25e-08, 8.99e-09, 6.25e-09, 4.44e-09, 3.45e-09], ['3', 9.32e-09, 8.60e-09, 5.67e-09, 3.68e-09, 2.53e-09, 1.75e-09]] df = pd.DataFrame(data, columns = ['id', '1','2','3','4','5','6']).set_index('id') # create figure and axis fig, ax = plt.subplots(figsize=(10, 10)) # setting the axis' labels ax.set_ylabel('Absolute Power (log)',fontsize=12) ax.set_xlabel('Frequencies',fontsize=12) # transposing (switchung rows and columns) of DataFrame df and # plot a line for each column on the axis ax, which was created previously df.T.plot(ax=ax)  The result looks like this:
Question: I'm trying to create a transformer that changes types of columns from "object" to "category", so I created custom class for that: from sklearn.base import BaseEstimator, TransformerMixin class ChangeToCategory(BaseEstimator, TransformerMixin): def __init__(self, to_categories = None): self.to_categories_ = to_categories def fit(self, X, y=None): return self def transform(self, X, y=None): X_ = X.copy() for cat in self.to_categories_: X_[cat] = X_[cat].astype("category") return X_  But I'm getting error when I try to create object of this class: ChangeToCategory(to_categories=["Sex", "Embarked"]) AttributeError: 'ChangeToCategory' object has no attribute 'to_categories'  What am I doing wrong? Based on error, I assume that it somewhere try to call attribute "to_categories", but I use attribute "to_categories_" - with underscore, and without is variable from init, which I don't call anywhere.

Answer: So, I found there I was wrong. From sklearn documentation you must initialize all estimator parameters as attributes of the class.  In addition, every keyword argument accepted by init should correspond to an attribute on the instance. Scikit-learn relies on this to find the relevant attributes to set on an estimator when doing model selection.
Question: I would like to do the following: given an instance of a Base class create an object of a Wrapper class that has all the methods and attributes of the Base class + some additional functionality. class Base: def __init__(self, *args, **kwargs): self.base_param_1 = ... # some stuff def base_method_1(self, *args, **kwargs): # some stuff class Wrapper(...): def __init__(self, cls_instance, *args, **kwargs): self.wrapper_param_1 = ... # some stuff def wrapper_method_1(self, *args, **kwargs): # some stuff  The use case is like the following: wrapper_obj = Wrapper(Base(*base_args, **base_kwargs), *wrapper_args, *wrapper_kwargs)  The expected behavior is that one can access base_param_1, wrapper_param_1 and base_param_1, base_param_2. It is important that the fields of the Base class are the same in Wrapper (no copying). I've seen that new functionality can be added in this way Adding a Method to an Existing Object Instance, but this approach has caveats and is not recommended. Inheritance seems not to be an option here, since I am an already constructed object and this Wrapper can take different classes, despite with common Base. EDIT It is also required that Wrapper object is identified as Base instance.

Answer: You can override __getattr__. That way, Wrapper specific attributes are looked up first, then the wrapped object's attributes are tried. class Wrapper: def __init__(self, base_obj, *args, **kwargs): self.base_obj = base_obj # some stuff def wrapper_method(self): return "new stuff" def __getattr__(self, name): return getattr(self.base_obj, name) w = Wrapper("abc") w.wrapper_method() # 'new stuff' w.upper() # 'ABC'
Question: I want to expand my data frame based on numeric values in two columns (index_start and index_end). My df looks like this: item index_start index_end A 1 3 B 4 7  I want this to expand to create rows for A from 1 to 3 and rows for B from 4 to 7 like so. item index_start index_end index A 1 3 1 A 1 3 2 A 1 3 3 B 4 7 4 B 4 7 5 B 4 7 6 B 4 7 7  Unsure how to implement this in Python/pandas.

Answer: You could use .explode() df['index'] = df.apply(lambda row: list(range(row['index_start'], row['index_end']+1)), axis=1) df.explode('index') item index_start index_end index 0 A 1 3 1 0 A 1 3 2 0 A 1 3 3 1 B 4 7 4 1 B 4 7 5 1 B 4 7 6 1 B 4 7 7
Question: In Python, you can create types dynamically using the function my_type = type(name, bases, dict). How would you specify a metaclass for this type my_type? (Ideally other than defining a throwaway class object that simply binds the metaclass to instantiated subclasses)

Answer: For Dynamic Type Creation where you need to provide keywords in the class statement (including, but not limited to, the keyword "metaclass"), you would use types.new_class. The following class definition: class A(B, C, metaclass=AMeta): pass  Can be created dynamically like: A = types.new_class( name="A", bases=(B, C), kwds={"metaclass": AMeta}, exec_body=None, )  This is preferable to directly calling type, or any other metaclass, since it resolves the metaclass bases - depending on the types of B and C, the type(A) here might not necessarily be AMeta.
Question: Recently I have been using the := operator in python quite a bit, in this way: if my_object := SomeClass.function_that_returns_object(): # do something with this object if it exists print(my_object.some_attribute)  The question Is there any way to do this in c++11 without the use of stdlib? for example in an arduino sketch if I wanted to use a method that may potentially return zero, such as: if(char * data = myFile.readBytes(data, dataLen)) { // do something } else { // do something else } 

Answer: Python's := assignment expression operator (aka, the "walrus" operator) returns the value of an assignment. C++'s = assignment operator (both copy assignment and move assignment, as well as other assignment operators) does essentially the same thing, but in a different way. The result of an assignment is a reference to the object that was assigned to, allowing that object to be evaluated in further expressions. So, the equivalent of: if my_object := SomeClass.function_that_returns_object(): # do something with this object if it exists print(my_object.some_attribute)  Would be just like you showed: SomeType *my_object; if ((my_object = SomeClass.function_that_returns_object())) { // do something with this object if it exists print(my_object->some_attribute); }  If function_that_returns_object() returns a null pointer, the if evaluates my_object as false, otherwise it evaluates as true. The same can be done with other types, eg: int value; if ((value = SomeClass.function_that_returns_int()) == 12345) { // do something with this value if it matches }
Question: My CS teacher told me that """ triple quotations are used as comments, yet I learned it as strings with line-breaks and indentations. This got me thinking - does python completely triple quote lines outside of relevant statements? """is this completely ignored like a comment"""  or, is the computer actually considering this?

Answer: Triple quoted strings are used as comment by many developers but it is actually not a comment, it is similar to regular strings in python but it allows the string to be in multi-line. You will find no official reference for triple quoted strings to be a comment. In python, there is only one type of comment that starts with hash # and can contain only a single line of text. According to PEP 257, it can however be used as a docstring, which is again not really a comment. def foo(): """ Developer friendly text for describing the purpose of function Some test cases used by different unit testing libraries """ ... # body of the function  You can just assign them to a variable as you do with single quoted strings: x = """a multi-line text enclosed by triple quotes """  Furthermore, if you try in repl, triple quoted strings get printed, had it really been a comment, should it have been printed?: >>> #comment >>> """triple quoted""" 'triple quoted'
Question: Is there any support to automatically implement all abstract methods of an abstract class in VSCode with Python Environment? class AbstractClass(ABC): @abstractclass def abstract_method(): pass class NonAbstractClass(AbstractClass): # shortcut in vscode to implement all abstract methods # it works if I start writing methods and then it autocompletes, that is not what I am looking for 

Answer: Seems like it is now (2024-04-17) in pre-release. Checkout the release notes.