eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: For the following df I wish to change the values in columns A,B and C to those in columns X,Y and Z, taking into account a boolean selection on column B. columns = {"A":[1,2,3], "B":[4,pd.NA,6], "C":[7,8,9], "X":[10,20,30], "Y":[40,50,60], "Z":[70,80,90]} df = pd.DataFrame(columns) df A B C X Y Z 0 1 4 7 10 40 70 1 2 <NA> 8 20 50 80 2 3 6 9 30 60 90  However when I try to do the value reassignment I end up with NULLS. df.loc[~(df["B"].isna()), ["A","B","C"]] = df.loc[~(df["B"].isna()), ["X","Y","Z"]] df A B C X Y Z 0 NaN NaN NaN 10 40 70 1 2.0 <NA> 8.0 20 50 80 2 NaN NaN NaN 30 60 90  My desired result is:  A B C X Y Z 0 10 40 70 10 40 70 1 2 <NA> 8 20 50 80 2 30 60 90 30 60 90  If I do the reassignment on a single column then I get my expected result: df.loc[~(df["B"].isna()), "A"] = df.loc[~(df["B"].isna()), "X"] df A B C X Y Z 0 10 4 7 10 40 70 1 2 <NA> 8 20 50 80 2 30 6 9 30 60 90  However I expected that I should be able to do this on multiple columns at once. Any ideas what I am missing here? Thanks

Answer: Your columns are not matching on the two sides of the assignment. Since you use a mask on both sides, don't perform index alignment and directly pass the values: m = ~df['B'].isna() df.loc[m, ['A', 'B', 'C']] = df.loc[m, ['X', 'Y', 'Z']].values  For your approach to work, you could also rename the columns. In this case you would not need the mask on both sides and could benefit from the index alignment: cols_left = ['A', 'B', 'C'] cols_right = ['X', 'Y', 'Z'] df.loc[~df['B'].isna(), cols_left] = df[cols_right].rename( columns=dict(zip(cols_right, cols_left)) )  Output:  A B C X Y Z 0 10 40 70 10 40 70 1 2 <NA> 8 20 50 80 2 30 60 90 30 60 90
Question: I'm testing a function that performs a bulk upsert using UpdateOne with bulk_write. In production (using the real MongoDB client) everything works fine, but when running tests with mongomock I get this error: app.mongodb.exceptions.CatalogException: failure in mongo repository function `batch_upsert_catalog_by_sku`: BulkOperationBuilder.add_update() got an unexpected keyword argument 'sort'  I don't pass any sort parameter in my code. Here's the relevant function: def batch_upsert_catalog_by_sku(self, items: List[CatalogBySkuWrite]) -> None: operations = [] current_time = datetime.datetime.now(datetime.timezone.utc) for item in items: update_fields = item.model_dump() update_fields["updated_at"] = current_time operations.append( UpdateOne( {"sku": item.sku, "chain_id": item.chain_id}, { "$set": update_fields, "$setOnInsert": {"created_at": current_time}, }, upsert=True, ) ) if operations: result = self.collection.bulk_write(operations) logger.info("Batch upsert completed", matched=result.matched_count, upserted=result.upserted_count, modified=result.modified_count)  Has anyone seen this error with mongomock? Is it a version issue or a bug, and what would be a good workaround? I'm using mongomock version 4.3.0 and pymongo version 4.11. Thanks!

Answer: This seem to be related to pymongo version 4.11. In 4.10.1, the method _add_to_bulk in the class UpdateOne here calls add_update this way:  def _add_to_bulk(self, bulkobj: _AgnosticBulk) -> None: """Add this operation to the _AsyncBulk/_Bulk instance `bulkobj`.""" bulkobj.add_update( self._filter, self._doc, False, bool(self._upsert), collation=validate_collation_or_none(self._collation), array_filters=self._array_filters, hint=self._hint, )  However, in the current release (4.11), here, it calls it this way:  def _add_to_bulk(self, bulkobj: _AgnosticBulk) -> None: """Add this operation to the _AsyncBulk/_Bulk instance `bulkobj`.""" bulkobj.add_update( self._filter, self._doc, False, bool(self._upsert), collation=validate_collation_or_none(self._collation), array_filters=self._array_filters, hint=self._hint, sort=self._sort, )  Latest version of mongomock does not seem to support the sort parameter, check here Forcing pymongo to version < 4.11 seem to solve the issue on my side (using Poetry: poetry add "pymongo<4.11") I have opened an issue in mongomock repository.
Question: I used the pandas library to manipulate the original data down to a simple 2 column csv file and rename it to a text file. The file has triple double quotes that I need replaced with single double quotes. Every line of the file is formatted as: """QA12345""","""Some Other Text"""  What I need is: "QA12345","Some Other Text"  This python snippet wipes out the file after it finishes. with fileinput.FileInput(input_file, inplace=True) as f: next(f) for line in f: line.replace("""""", '"')  It doesn't work with line.replace('"""', '"')  either. I've also tried adjusting the input values to be '"Some Other Text"' and variations (""" and '\"') but nothing seems to work. I believe the triple quote is causing the issue, but I don't know what I have to do to.

Answer: Judging by the OP's code, I guess the aim are  Edit file in-place keep the first line unmodified For the rest of the lines, replace 3 double quotes with a single one  My solution is almost the same, but with print(): with fileinput.input(input_file, inplace=True) as stream: print(next(stream), end="") for line in stream: print(line.replace('"""', '"'), end="")  That should give the desired result.
Question: I am trying to access raw 990 (nonprofit tax returns) XML data through the AWS datalake. The XML files are organized into Zip files split by the month in which the IRS processed them ("2024_1A" for January, "2024_2A" for Feburary). See photo for the Zip files I am iterating through. In certain months, there are multiple Zip files where there were too many returns for a single Zip file.  I have written a Python script that is able to access the Datalake and process the XML files for 10 out of 14 Zip files. However, for the very large Zips -- 5A, 5B, 11A, 11B -- it returns this error: "Error processing file EfileData/XmlZips/2024_TEOS_XML_05A.zip: That compression method is not supported" The only differentiating factor between these Zips and the other Zips seems to be their size -- see the fixes/checks I've tried below. This is the code I'm using right now to unzip the files to a temporary directory. Does anyone have thoughts on why it would work for all the other zip files but not 5A, 5B, 11A, 11B, and how I can fix it? Thank you! with zipfile.ZipFile(local_zip_path, 'r') as zip_ref: zip_ref.extractall(temp_dir)  Here's some of the fixes I've tried:  Using the pyz7r library and the 7zip tool through Python Confirming the Zips are compression type 9, which corresponds to DEFLATE compression (zipfile.ZIP_DEFLATED) which should be supported by Python's zipfile module. Downloaded the Zip files onto my computer and previewed the .XML files inside to confirm they're not corrupted. Tried unzipping the Zip file onto my computer to then upload directly into the Google Colab workspace where I am working, but it was too large (70,000 files in each) and it took several hours to download just one of the 4 Zip files and it froze google chrome each time I tried to them upload it into Google colab. Attempted to extract the Zip file in batches to be more efficient 

Answer: Compression method 9 is not Deflate. (Deflate is method 8.) Method 9 is a proprietary PKWare enhancement of Deflate called Deflate64. Python's zipfile does not support it. You would need to use an unzip utility, such as Info-ZIP's unzip, 7-zip, or the like. When I try it, zipfile raises an error that says exactly that, which you are also seeing: raise NotImplementedError("That compression method is not supported")  R1D3R175 notes in a comment below that there exists a zipfile-deflate64 project that can handle the Deflate64 method. These zip files were likely made using Windows' built-in compression tool, which elects to use Deflate64 when the sum of the file sizes is greater than 2 GB.
Question: I have columns of data in an excel sheet. The data is paired. I want to plot column 1 and 2 together, 3 and 4 together and so on. In my case I have four columns. I've tried creating a for loop to go through the data and plot it but I keep getting a separate plots. Anyone know what's wrong here?  for i in range(0,data.shape[1]): if i%2 == 0: plt.plot(data.iloc[:,i], data.iloc[:,i+1]) plt.xlabel('Strain (mm)') plt.ylabel('Stress (lbf)') plt.show() 

Answer: If I understand you correctly, you want to generate a single plot using several column combinations. You should create an axis object and then add the data to it. import numpy as np import pandas as pd from matplotlib import pyplot as plt data = pd.DataFrame({ 'col1': np.linspace(0, 0.2, 100), 'col2': np.linspace(0, 0.2, 100), 'col3': np.linspace(0.2, 1, 100), 'col4': np.linspace(0.2, 1, 100) }) fig, ax = plt.subplots() for i in range(0, data.shape[1]): if i%2 == 0: ax.plot(data.iloc[:,i], data.iloc[:,i+1]) ax.set_xlabel('Strain (mm)') ax.set_ylabel('Stress (lbf)')
Question: I'm relatively new to using polars and it seems to be very verbose compared to pandas for what I would consider even relatively basic manipulations. Case in point, the shortest way I could figure out doing a softmax over a lazy dataframe is the following: import polars as pl data = pl.DataFrame({'a': [1,2,3,4,5,6,7,8,9,10], 'b':[5,5,5,5,5,5,5,5,5,5], 'c': [10,9,8,7,6,5,4,3,2,1]}).lazy() cols = ['a','b','c'] data = data.with_columns([ pl.col(c).exp().alias(c) for c in cols]) # Exp all columns data = data.with_columns(pl.sum_horizontal(cols).alias('sum')) # Get row sum of exps data = data.with_columns([ (pl.col(c)/pl.col('sum')).alias(c) for c in cols ]).drop('sum') data.collect()  Am I missing something and is there a shorter, more readable way of achieving this?

Answer: You would use a multi-col selection e.g. pl.all() instead of list comprehensions. (Or pl.col(cols) for a named "subset" of columns) df.with_columns( pl.all().exp() / pl.sum_horizontal(pl.all().exp()) )  shape: (10, 3) ┌──────────┬──────────┬──────────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ f64 │ ╞══════════╪══════════╪══════════╡ │ 0.000123 ┆ 0.006692 ┆ 0.993185 │ │ 0.000895 ┆ 0.01797 ┆ 0.981135 │ │ 0.006377 ┆ 0.047123 ┆ 0.946499 │ │ 0.04201 ┆ 0.114195 ┆ 0.843795 │ │ 0.211942 ┆ 0.211942 ┆ 0.576117 │ │ 0.576117 ┆ 0.211942 ┆ 0.211942 │ │ 0.843795 ┆ 0.114195 ┆ 0.04201 │ │ 0.946499 ┆ 0.047123 ┆ 0.006377 │ │ 0.981135 ┆ 0.01797 ┆ 0.000895 │ │ 0.993185 ┆ 0.006692 ┆ 0.000123 │ └──────────┴──────────┴──────────┘  With LazyFrames we can use .explain() to inspect the query plan. plan = df.lazy().with_columns(pl.all().exp() / pl.sum_horizontal(pl.all().exp())).explain() print(plan)  # simple π 3/7 ["a", "b", "c"] # WITH_COLUMNS: # [[(col("__POLARS_CSER_0x9b1b3182d015f390")) / (col("__POLARS_CSER_0x762bfea120ea9e6"))].alias("a"), [(col("__POLARS_CSER_0xb82f49f764da7a09")) / (col("__POLARS_CSER_0x762bfea120ea9e6"))].alias("b"), [(col("__POLARS_CSER_0x1a200912e2bcc700")) / (col("__POLARS_CSER_0x762bfea120ea9e6"))].alias("c")] # WITH_COLUMNS: # [col("a").exp().alias("__POLARS_CSER_0x9b1b3182d015f390"), col("b").exp().alias("__POLARS_CSER_0xb82f49f764da7a09"), col("c").exp().alias("__POLARS_CSER_0x1a200912e2bcc700"), col("a").exp().sum_horizontal([col("b").exp(), col("c").exp()]).alias("__POLARS_CSER_0x762bfea120ea9e6")] # DF ["a", "b", "c"]; PROJECT */3 COLUMNS  Polars caches the duplicate pl.all().exp() expression into a temp __POLARS_CSER* column for you. See also:  https://docs.pola.rs/user-guide/lazy/optimizations/
Question: This script: import numpy as np a = np.array([2, 3, 1, 9], dtype='i4') print(a) print(f'{a=}')  produces: [2 3 1 9] a=array([2, 3, 1, 9], dtype=int32)  Is there a way to get just a=[2 3 1 9] from {a=} f-string as in the standard formatting in the first line?

Answer: By default, = f-string syntax calls repr on the thing to be formatted, since that's usually more useful than str or format for the debugging use cases the = syntax was designed for. You can have it apply normal formatting logic instead by specifying a format - even an empty format will do, so this would work: print(f'{a=:}')  or you can explicitly say to call str with !s: print(f'{a=!s}')
Question: Is it possible to access the object instance in a Python @classmethod annotated function when the call occurs via the instance itself rather than the class? class Foo(object): def __init__(self, text): self.text = text @classmethod def bar(cls): return None print(Foo.bar()) foo = Foo('foo') # If possible, I'd like to return "foo" print(foo.bar())  I'm working around limitations of a library which calls a class method via the instance (rather than the class) and was hoping I could somehow workaround the fact that I can only access the class in bar(cls) (I'd to access the instance to access .text). I don't think that's possible, but I thought I'd ask (I couldn't unearth something on the internet either for such a niche request).

Answer: One possible approach without modifying Foo is to define bar as an instance method of a subclass, but use a custom descriptor to make attribute access to the method return the class method of the same name of the super class instead when the attribute is accessed from the class, when the instance passed to __get__ is None: class hybridmethod: def __init__(self, f): self.f = f def __get__(self, obj, cls=None): if obj is not None: return self.f.__get__(obj) return getattr(super(cls, cls), self.f.__name__).__get__(cls) class MyFoo(Foo): @hybridmethod def bar(self): return self.text print(MyFoo.bar()) # outputs None foo = MyFoo('foo') print(foo.bar()) # outputs foo  Demo: https://ideone.com/yVAzr3 EDIT: Since you have now clarified in the comment that you have full control over Foo, you can instead modify Foo.bar directly such that it is a custom descriptor whose __get__ method calls a class method or an instance method based on the aforementioned condition. For convenience I've made the descriptor a decorator that decorates a class method with an additional instancemethod method that can decorate an instance method: class instanceable_classmethod(classmethod): def __init__(self, func): super().__init__(func) self.instance_func = self.__func__ def instancemethod(self, func): self.instance_func = func return self def __get__(self, obj, cls=None): if obj is None: return super().__get__(obj, cls) return self.instance_func.__get__(obj) class Foo(object): def __init__(self, text): self.text = text @instanceable_classmethod def bar(cls): return None @bar.instancemethod def bar(self): return self.text print(Foo.bar()) # outputs None foo = Foo('foo') print(foo.bar()) # outputs foo  Demo: https://ideone.com/PmSnUC
Question: I am trying to compile translation messages in my Django project by running the following command: python manage.py compilemessages  However, I get this error: CommandError: Can't find msgfmt. Make sure you have GNU gettext tools 0.15 or newer installed.  So then I tried installing gettext. sudo apt update && sudo apt install gettext -y  After installation, I still get the same error when running python manage.py compilemessages. Checked if msgfmt is installed by msgfmt --version but it says: command not found OS is Ubuntu and Django version is 4.2.5. How can I resolve this issue and make Django recognize msgfmt? Any help would be appreciated!

Answer: I've got the point. After deactivating the virtual environment, I tried installing gettext. Now it is okay.
Question: Let's assume that I have a generic class like that : import typing type_variable = typing.TypeVar("type_variable") class OneClass(typing.Generic[type_variable]): def __init__(self, value: type_variable): self.value = value  Now I create two simple objects : a = OneClass("a str-typped object") b = OneClass(b'a byte-typped object')  When I check if a and b have same type using type, it says True : >>> type(a) == type(b) True  When I check if a is of type OneClass[str], it says False: >>> type(a) == OneClass[str] False  I know that there I can use something like type(a.value) == type(b.value), but in some other cases I can have this kind of objects : tuple_object_1 = OneClass(("a", "first", "tupple", "with", "strings")) tuple_object_2 = OneClass((b'a', b'second', b'tupple', b'with', b'bytes'))  In that case I have as types :  tuple_object_1 : OneClass[tuple[str]] tuple_object_2 : OneClass[tuple[bytes]]  and so evaluating the types of tuple_object_1 and tuple_object_2 only returns tuple and not something like tuple[str] and tuple[bytes], so the type evaluation of the content is also not valid. How can I get the type of those kind of objects, considering also their generic types? Something like a deep type. Thanks for help!

Answer: You are making a fundamental error. You are confusing types (int, float, OneClass) with type annotations (all the previous types, as well as OneClass[int], tuple[int, ...], etc.) As far as Python is concerned, tuple_object_1 and tuple_object_2 are just tuples, nothing more. It doesn't care in the least what the types of the elements are within the tuple until you actually access them. Type annotations are used by IDEs and checkers to verify the correctness of your code. If you declare something to be List[int], it will be your IDE or your lint checker that warns you if you try to add a string to it. But Python couldn't care less.
Question: I have a Polars LazyFrame and would like to add to it columns from another LazyFrame. The two LazyFrames have the same number of rows and different columns. I have tried the following, which doesn't work as with_columns expects an iterable. def append_columns(df:pl.LazyFrame): df2 = pl.LazyFrame([1,2]) return df.with_columns(df2) 

Answer: For this, pl.concat setting how="horizontal" might be used. import polars as pl df = pl.LazyFrame({ "a": [1, 2, 3], "b": [4, 5, 6], }) other = pl.LazyFrame({ "c": [9, 10, 11], "d": [12, 13, 14], "e": [15, 16, 17], }) result = pl.concat((df, other.select("c", "d")), how="horizontal")  The resulting pl.LazyFrame then looks as follows. shape: (3, 4) ┌─────┬─────┬─────┬─────┐ │ a ┆ b ┆ c ┆ d │ │ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 │ ╞═════╪═════╪═════╪═════╡ │ 1 ┆ 4 ┆ 9 ┆ 12 │ │ 2 ┆ 5 ┆ 10 ┆ 13 │ │ 3 ┆ 6 ┆ 11 ┆ 14 │ └─────┴─────┴─────┴─────┘
Question: I have an application that takes an input spreadsheet filled in by the user. I'm trying to bug fix, and I've just noticed that in one True/False column, they've written FLASE instead of FALSE. I'm trying to write in as many workarounds for user error as I can, as the users of this app aren't very technical, so I was wondering if there's a way to convert this column to type bool, but set any typos to False? I appreciate this would also set something like TURE to be false too. For example: df = pd.DataFrame({'bool_col':['True', 'Flase', 'False', 'True'], 'foo':[1,2,3,4]})  Running df['bool_col'] = df['bool_col'].astype(bool) returns True for everything (as they're all non-empty strings), however I would like it to reutrn True, False, False, True.

Answer: If you want a generic approach, you could use fuzzy matching, for example with thefuzz: from thefuzz import process def to_bool(s, threshold=60): bools = [True, False] choices = list(map(str, bools)) match, score = process.extractOne(s, choices) d = dict(zip(choices, bools)) if score > threshold: return d[match] df['bool'] = df['bool_col'].map(to_bool)  Example:  bool_col foo bool 0 True 1 True 1 Flase 2 False 2 False 3 False 3 True 4 True 4 ture 5 True 5 banana 6 None
Question: This is a part of my code. For example, even if 3 out of 10 ends first, I want the next 3 to start right away and always keep 10 threads running. However, the current code is moving on to the next 10 only when all 10 are completely finished. How can I modify the code? I always want to keep a certain number of threads, but as it stands, I have to finish the previous 10 threads completely before moving on to the next 10. import concurrent.futures import time def example_task(n): print(f"Task {n} started.") time.sleep(n) print(f"Task {n} completed.") return n with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor: futures = [] for i in range(10): futures.append(executor.submit(example_task, i+1)) for future in concurrent.futures.as_completed(futures): try: result = future.result() print(f"Result of task: {result}") next_task = len(futures) + 1 futures.append(executor.submit(example_task, next_task)) except Exception as e: print(f"Error: {e}") 

Answer: Use a queue-based approach with ThreadPoolExecutor, where tasks are continuously submitted as soon as one completes. import concurrent.futures import time import itertools def example_task(n): print(f"Task {n} started.") time.sleep(n) # Simulate work print(f"Task {n} completed.") return n def main(): max_threads = 5 total_tasks = 20 # Total number of tasks you want to process task_counter = itertools.count(1) # Infinite counter for task numbers with concurrent.futures.ThreadPoolExecutor(max_workers=max_threads) as executor: futures = {} # Submit initial batch of tasks for _ in range(max_threads): task_id = next(task_counter) futures[executor.submit(example_task, task_id)] = task_id # Process tasks dynamically while futures: done, _ = concurrent.futures.wait(futures, return_when=concurrent.futures.FIRST_COMPLETED) for future in done: task_id = futures.pop(future) # Remove completed task try: result = future.result() print(f"Result of task {result}") # Submit a new task if we haven't reached the total task limit if task_id < total_tasks: new_task_id = next(task_counter) futures[executor.submit(example_task, new_task_id)] = new_task_id else: print("FINISHING") break except Exception as e: print(f"Task {task_id} failed: {e}") if __name__ == "__main__": main()
Question: I am using SeleniumBase in CDP Mode. I am having a hard time figuring out if this a python issue or SeleniumBase issue. The below simple example shows my problem: from seleniumbase import SB with SB(uc=True, locale_code="en", headless=True) as sb: link = "https://news.ycombinator.com" print(f"\nOpening {link}") sb.wait_for_ready_state_complete(timeout=120) sb.activate_cdp_mode(link) script = f""" function getSomeValue() {{ return '42'; }} return getSomeValue(); """ # data = sb.execute_script(script) data = sb.cdp.evaluate(script) print(data) print("Finished!")  This throws error: seleniumbase.undetected.cdp_driver.connection.ProtocolException: exceptionId: 1 text: Uncaught lineNumber: 5 columnNumber: 4 scriptId: 6 exception: type: object subtype: error className: SyntaxError description: SyntaxError: Illegal return statement objectId: 3089353218542582072.1.2  Notice above that I have tried both sb.execute_script(script) and sb.cdp.evaluate(script) and both give the same issue. How can I execute such scripts?

Answer: In CDP Mode, don't include the final return when evaluating JS. Your script part should look like this: script = f""" function getSomeValue() {{ return '42'; }} getSomeValue(); """ data = sb.cdp.evaluate(script)  (Instead of using "return getSomeValue();", which breaks evaluate(expression).)
Question: import re texto = "ABC ABC. ABC.png ABC thumb.png" regex = r"ABC(?!.png)|ABC(?! thumb.png)" novo = re.sub(regex, "bueno", texto) print(novo)  I'm trying to replace the ABC word with exceptions. I only want to replace it if it doesn't follow the word ".png" or " thumb.png". The string would be then "ABC thumb.png" I expected bueno bueno. ABC.png ABC thumb.png  But the output is this bueno bueno. bueno.png bueno thumb.png  It isn't detecting the space and it actually messes up the first condition.

Answer: Starting with your original pattern: ABC(?!\.png)|ABC(?! thumb\.png) (Note: Dot is a regex metacharacter and should be escaped with backslash)  This will match ABC which is not followed by .png or ABC not followed by  thumb.png. Every possible occurrence of ABC will match this pattern. Therefore, all occurrences of ABC will be match, because every extension will match at least one of the two conditions. We can write the following correction: \bABC(?!\.png| thumb\.png)  This pattern says to match:  \b word boundary ABC match ABC (?!\.png| thumb\.png) neither .png or  thumb.png follows  The negative lookahead used here basically has AND flavored logic, and will exclude both following extensions.