eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm currently getting into proxy models and I actually cannot understand when we should give them respect. For me they look very similar to managers Is there some differences or we can implement the same things using proxy models and managers?

Answer: Django managers are classes that manage the database query operations on a particular model. Django model manager Proxy models allow you to create a new model that inherits from an existing model but does not create a new database table. proxy model Let me give you an example: class Book(models.Model): title = models.CharField(max_length=100) author = models.CharField(max_length=100) published_date = models.DateField() is_downloadable = models.BooleanField(default=True)  If this is your model manager: class BookMnanger(models.Manager): def by_link(self): return self.filter(is_downloadable=True)  and you new manager: class Book(models.Model): title = models.CharField(max_length=100) author = models.CharField(max_length=100) published_date = models.DateField() is_downloadable = models.BooleanField(default=True) # your new manager should register to your model downloading = BookMnanger()  now, your new custom manager can work as below: my_books = Book.downloading.all() print(my_books)  but the proxy: class BookProxy(Book): class Meta: proxy = True def special_method(self): return f"{self.title} by {self.author}, published on {self.published_date}"  and your proxy can work like this: book = BookProxy.objects.first() print(book.special_method())  proxies are the way to change behavior of your model but, managers will change your specific queries I can give you more link about them, if you need?
Question: I'm trying to account for room usage only during business hours and abridge an event duration if it runs past the end of business hours. I have a dataframe like this: import polars as pl from datetime import datetime df = pl.DataFrame({ 'name': 'foo', 'start': datetime.fromisoformat('2025-01-01 08:00:00'), 'end': datetime.fromisoformat('2025-01-01 18:00:00'), # ends after business hours 'business_end': datetime.fromisoformat('2025-01-01 17:00:00') })  I want to create a duration column that is equal to end unless it's after business_end otherwise set to business_end. For this, I tried the following: df.with_columns( duration=pl.col("end") - pl.col("start") if pl.col("end") <= pl.col("business_end") else pl.col("business_end") - pl.col("start") )  This gives an error:  TypeError: the truth value of an Expr is ambiguous  Thoughts about how to produce the desired row from the conditional? I can use filter() to find rows where event ends are after business ends, create a frame of those, replace the end time value, merge back in, etc. but I was hoping to keep the original data and only add a new column.

Answer: Short answer You use when/then/otherwise instead of if else  df.with_columns( duration=pl.when(pl.col("end") <= pl.col("business_end")) .then(pl.col("end") - pl.col("start")) .otherwise(pl.col("business_end") - pl.col("start")) )  Background polars works with expressions inside contexts. What's that mean? Contexts are your with_columns, select, group_by, agg, etc. The inputs to contexts are expressions. Expressions usually start with pl.col() or pl.lit(). They have lots of methods which also return expressions which makes them chainable. The thing about expressions is that they don't have values, they're just instructions. One way to see that clearly is to assign an expression to a normal variable like end=pl.col("end"). You can do that without any DataFrames existing. Once you have a df, you can use that expr in its context df.select(end). When the select context gets the expression pl.col("end"), that's when it'll go fetch the column. You could also make a more complicated expression like my_sum = (pl.col("a") * 2 + pl.col("b").pow(3)) and then even chain off of it df.select(my_sum*2+5) Now getting back to the if, because pl.col("end") doesn't have any values associated with it, python can't evaluate if pl.col("end") <= pl.col("other") which is why you're getting that error. python doesn't have an overload for if so you just can't use it inside a context. Instead you can use the when then otherwise construct.
Question: By grouping all the items within an array into pairs and getting their absolute differences, what is the minimum sum of their absolute differences? Example: [4, 1, 2, 3] should return 2 because |1 - 2| + |3 - 4| = 2 [1, 3, 3, 4, 5] should return 1 because |3 - 3| + |4 - 5| = 1  Getting the result for arrays with an even length is pretty easy, because it just requires sorting the array and getting the differences between numbers right next to each other. However, arrays with an odd length are pretty hard because the addition of an extra number. Thus, my current solution for odd length arrays is to remove each number and check the sum using the even-length array approach. Current solution: def smallest_sum(arr): arr = sorted(arr) if len(arr) % 2 == 0: return even_arr_sum(arr) else: return odd_arr_sum(arr) def even_arr_sum(arr): total = 0 for i in range(0, len(arr), 2): total += arr[i+1] - arr[i] return total def odd_arr_sum(arr): total = 0 for i in range(len(arr)): s = even_arr_sum(arr[:i] + arr[i+1:]) if total == 0 or s < total: total = s return total assert smallest_sum([4, 1, 2, 3]) == 2 assert smallest_sum([1, 3, 3, 4, 5]) == 1  However, this is extremely slow and gives a time of n2. Is there a smarter way to handle arrays with an odd length?

Answer: Imagine you have sorted seven numbers A to G and you leave out A, thus calculating (C-B)+(E-D)+(G-F). That's adding or subtracting them like this: A B C D E F G - + - + - + (leaving out A)  And this is how it looks in general, for leaving out each of the numbers: A B C D E F G - + - + - + (leaving out A) - + - + - + (leaving out B) - + - + - + ... - + - + - + - + - + - + - + - + - + - + - + - +  From one row to the next, there's little change. So first compute the total for leaving out A. Then for the total for leaving out B, simply subtract A and add B. And so on.
Question: Currently working on a classifier using PyWavelets, here is my calculation block: class WaveletLayer(nn.Module): def __init__(self): super(WaveletLayer, self).__init__() def forward(self, x): def wavelet_transform(img): coeffs = pywt.dwt2(img.cpu().numpy(), "haar") LL, (LH, HL, HH) = coeffs return ( torch.from_numpy(LL).to(img.device), torch.from_numpy(LH).to(img.device), torch.from_numpy(HL).to(img.device), torch.from_numpy(HH).to(img.device), ) # Apply wavelet transform to each channel separately LL, LH, HL, HH = zip( *[wavelet_transform(x[:, i : i + 1]) for i in range(x.shape[1])] ) # Concatenate the results LL = torch.cat(LL, dim=1) LH = torch.cat(LH, dim=1) HL = torch.cat(HL, dim=1) HH = torch.cat(HH, dim=1) return torch.cat([LL, LH, HL, HH], dim=1)  The output from this module goes to a resnet block for learning, while doing this I find my CPU clogged and thus slowing down my training process I am trying to use the GPUs for these calculations.

Answer: Since you only seem to be interested in the Haar wavelet, you can pretty much implement it yourself:  The high-frequency component of the Haar wavelet along each dimension can be written as a pairwise difference. The low-frequency component of the Haar wavelet along each dimension can be written as a pairwise sum.  The following code achieves this in pure PyTorch: class HaarWaveletLayer(nn.Module): def l_0(self, t): # sum ("low") along cols t = torch.cat([t, t[..., -1:, :]], dim=-2) if t.shape[-2] % 2 else t return (t[..., ::2, :] + t[..., 1::2, :]) def l_1(self, t): # sum ("low") along rows t = torch.cat([t, t[..., :, -1:]], dim=-1) if t.shape[-1] % 2 else t return (t[..., :, ::2] + t[..., :, 1::2]) def h_0(self, t): # diff ("hi") along cols t = torch.cat([t, t[..., -1:, :]], dim=-2) if t.shape[-2] % 2 else t return (t[..., ::2, :] - t[..., 1::2, :]) def h_1(self, t): # diff ("hi") along rows t = torch.cat([t, t[..., :, -1:]], dim=-1) if t.shape[-1] % 2 else t return (t[..., :, ::2] - t[..., :, 1::2]) def forward(self, x): x = .5 * x l_1 = self.l_1(x) h_1 = self.h_1(x) ll = self.l_0(l_1) lh = self.h_0(l_1) hl = self.l_0(h_1) hh = self.h_0(h_1) return torch.cat([ll, lh, hl, hh], dim=1)  In combination with your given code, you can convince yourself of the equivalence as follows: t = torch.rand((7, 3, 127, 128)).to("cuda:0") result_given = WaveletLayer()(t) result_proposed = HaarWaveletLayer()(t) # Same result? assert (result_given - result_proposed).abs().max() < 1e-5 # Time comparison from timeit import Timer num_timings = 100 print("time given: ", Timer(lambda: WaveletLayer()(t)).timeit(num_timings)) print("time proposed:", Timer(lambda: HaarWaveletLayer()(t)).timeit(num_timings))  The timing shows a speedup of more than a factor of 10 on my machine. Notes  The t = torch.cat... parts are only necessary if you want to be able to handle odd-shaped images: In that case, we pad by replicating the last row and column, respectively, mimicking the default padding of PyWavelets. Multiplying x with .5 is done for normalization. Compare this discussion on the Signal Processing Stack Exchange for more details.
Question: I'm running into a situation I don't know what to do: The data is a list, no index. Sample data: data = [ {'fields': ['2024-10-07T21:22:01', 'USER-A', 21, 0, 0, 21]}, {'fields': ['2024-10-07T21:18:28', 'USER-B', 20, 20, 0, 0, 0, 45]} ]  The column header is in another: cols = ['Created On', 'Created By', 'Transaction Count (ALL)', 'X Pending', 'X Cancelled (X)', 'X Completed (Y)']  I have tried using pandas.DataFrame as well as json_normalize, I either get a single column table with each value as a row, or I got all values as a column, and when I try with using "fields", it tells me "list indices must be integers or slices, not str" which I don't understand why I get this... what is the best way to have these info into a dataframe please? (the number of data elements and number of column headers may not be consistent just for example sake, the real data has things aligned)

Answer: You could combine two DataFrame constructors: data = [{'fields': ['2024-10-07T21:22:01', 'USER-A', 21, 0, 0, 21]}, {'fields': ['2024-10-07T21:18:28', 'USER-B', 20, 20, 0, 0, 0, 45]}, ] out = pd.DataFrame(pd.DataFrame(data)['fields'].tolist())  Output:  0 1 2 3 4 5 6 7 0 2024-10-07T21:22:01 USER-A 21 0 0 21 NaN NaN 1 2024-10-07T21:18:28 USER-B 20 20 0 0 0.0 45.0  If you also have a list of columns cols, you could truncate the columns: cols = ['Created On', 'Created By', 'Transaction Count (ALL)', 'X Pending', 'X Cancelled (X)', 'X Completed (Y)'] out = pd.DataFrame(pd.DataFrame(data)['fields'].str[:len(cols)].tolist(), columns=cols)  Output:  Created On Created By Transaction Count (ALL) X Pending X Cancelled (X) X Completed (Y) 0 2024-10-07T21:22:01 USER-A 21 0 0 21 1 2024-10-07T21:18:28 USER-B 20 20 0 0  Or rename to keep the extra columns: out = (pd.DataFrame(pd.DataFrame(data)['fields'].tolist()) .rename(columns=dict(enumerate(cols))) )  Output:  Created On Created By Transaction Count (ALL) X Pending X Cancelled (X) X Completed (Y) 6 7 0 2024-10-07T21:22:01 USER-A 21 0 0 21 NaN NaN 1 2024-10-07T21:18:28 USER-B 20 20 0 0 0.0 45.0  But, honestly, better pre-process in pure python, this will be more efficient/explicit: # truncation out = pd.DataFrame((dict(zip(cols, d['fields'])) for d in data)) # alternative truncation out = pd.DataFrame([d['fields'][:len(cols)] for d in data], columns=cols) # renaming out = (pd.DataFrame([d['fields'] for d in data]) .rename(columns=dict(enumerate(cols))) )
Question: I have a Pandas dataframe in wide format that looks like this: import pandas as pd df = pd.DataFrame({'Class_ID': {0: 432, 1: 493, 2: 32}, 'f_proba_1': {0: 3, 1: 8, 2: 6}, 'f_proba_2': {0: 4, 1: 9, 2: 9}, 'f_proba_3': {0: 2, 1: 4, 2: 1}, 'p_proba_1': {0: 3, 1: 82, 2: 36}, 'p_proba_2': {0: 2, 1: 92, 2: 96}, 'p_proba_3': {0: 8, 1: 41, 2: 18}, 'Meeting_ID': {0: 27, 1: 23, 2: 21}}) df Class_ID f_proba_1 f_proba_2 f_proba_3 p_proba_1 p_proba_2 p_proba_3 Meeting_ID 0 432 3 4 2 3 2 8 27 1 493 8 9 4 82 92 41 23 2 32 6 9 1 36 96 18 21  and I would like to convert to long format:  Class_ID Student_ID f_proba p_proba Meeting_ID 0 432 1 3 3 27 1 432 2 4 2 27 2 432 3 2 8 27 3 493 1 8 82 23 4 493 2 9 92 23 5 493 3 4 41 23 6 32 1 6 36 21 7 32 2 9 96 21 8 32 3 1 18 21  So I have tried .melt in Pandas and here is my code out = pd.melt(df, id_vars = ['Class_ID', 'Meeting_ID'], value_vars = ['f_proba_1','f_proba_2','f_proba_3','p_proba_1','p_proba_2','p_proba_3'], var_name = 'Student_ID', value_name = ['f_proba', 'p_proba']) out  but it didn't work.

Answer: You can use pd.wide_to_long for this: out = (pd.wide_to_long(df, stubnames=['f_proba', 'p_proba'], i=['Class_ID', 'Meeting_ID'], j='Student_ID', sep='_') .reset_index() )  Output:  Class_ID Meeting_ID Student_ID f_proba p_proba 0 432 27 1 3 3 1 432 27 2 4 2 2 432 27 3 2 8 3 493 23 1 8 82 4 493 23 2 9 92 5 493 23 3 4 41 6 32 21 1 6 36 7 32 21 2 9 96 8 32 21 3 1 18
Question: How can I save a tf.data.Dataset in multiple shards using tf.data.Dataset.save()? I am reading in my dataset from CSV using tf.data.experimental.make_csv_dataset. The TF docs here are not very helpful. There is a shard_func argument, but the examples given aren't helpfull and its not clear how to map to an int in a deterministic way. Using random ints doesn't seem to work either. The solution in a similar question here generates an error for me TypeError: unsupported operand type(s) for %: 'collections.OrderedDict' and 'int' Single Shard (works) The below successfully saves to a single shard. import pandas as pd import numpy as np import tensorflow as tf # gen data n=10000 pd.DataFrame( {'label': np.random.randint(low=0, high=2, size=n), 'f1': np.random.random(n), 'f2': np.random.random(n), 'f3': np.random.random(n), 'c1': np.random.randint(n), 'c2': np.random.randint(n)} ).to_csv('tmp.csv') # load data into a tf.data.Dataset data_ts = tf.data.experimental.make_csv_dataset( 'tmp.csv', 1, label_name='label', num_epochs=1) data_ts.save('tmp.data') # single shard, works!  Multiple shards usind randint (saves single shard) Trying to save to multiple shard using a random number, still only saves to a single shard, albeit with a random int in the file name. # Try sharding, using random numbers. def random_shard_function(features, label): return np.int64(np.random.randint(10)) data_ts.save('tmp2.data', shard_func=random_shard_function)   Modulo shard (error) Trying the sollution from this question. def modulo_shard_function(features, label): return x & 10 data_ts.save('tmp2.data', shard_func=modulo_shard_function)  TypeError: unsupported operand type(s) for &: 'collections.OrderedDict' and 'int' Debugging - no idea how shard_fun works. If I print out the inputs, it seems that the shard func is only run once, and the tensors are SymbolicTensors def debug_shard_function(features, label): for val in features.items(): print(f'{val=}') print(f'{label=}') print(f'{type(val[1])}') return np.int64(10) data_ts.save('tmp2.data', shard_func=debug_shard_function)  Output: Still saves to a single shard val=('', <tf.Tensor 'args_0:0' shape=(None,) dtype=int32>) val=('f1', <tf.Tensor 'args_3:0' shape=(None,) dtype=float32>) val=('f2', <tf.Tensor 'args_4:0' shape=(None,) dtype=float32>) val=('f3', <tf.Tensor 'args_5:0' shape=(None,) dtype=float32>) val=('c1', <tf.Tensor 'args_1:0' shape=(None,) dtype=int32>) val=('c2', <tf.Tensor 'args_2:0' shape=(None,) dtype=int32>) label=<tf.Tensor 'args_6:0' shape=(None,) dtype=int32> <class 'tensorflow.python.framework.ops.SymbolicTensor'> 

Answer: shard_func must return a scalar Tensor of type tf.int64 (not Python or NumPy integer). So you cannot just return np.int64(...) or do a Python‐level % on dictionary.You need to pick (or compute) tensor inside dataset element and return tf.cast(..., tf.int64). For example if your CSV has a column "c1" you could do: def shard_func(features, label): return tf.cast(features['c1'][0] % 10, tf.int64) data_ts.save("my_data",shard_func=shard_func)  This will produce up to 10 different shards (files) named my_data_0, my_data_1, etc
Question: I am encountering an AttributeError while fitting an XGBRegressor using RandomizedSearchCV from Scikit-learn. The error message states: 'super' object has no attribute '__sklearn_tags__'.  This occurs when I invoke the fit method on the RandomizedSearchCV object. I suspect it could be related to compatibility issues between Scikit-learn and XGBoost or Python version. I am using Python 3.12, and both Scikit-learn and XGBoost are installed with their latest versions. I attempted to tune the hyperparameters of an XGBRegressor using RandomizedSearchCV from Scikit-learn. I expected the model to fit the training data without issues and provide the best parameters after cross-validation. I also checked for compatibility issues, ensured the libraries were up-to-date, and reinstalled Scikit-learn and XGBoost, but the error persists.

Answer: Scikit-learn version 1.6 modified the API around its "tags", and that's the cause of this error. XGBoost has made the necessary changes in PR11021, but at present that hasn't made it into a released version. You can either keep your sklearn version <1.6, or build XGBoost directly from github (or upgrade XGBoost, after a new version is released). In sklearn 1.6.1, the error was downgraded to a warning (to be returned to an error in 1.7). So you may also install sklearn >=1.6.1,<1.7 and just expect DeprecationWarnings. See also sklearn Issue#30479 and 1.6.1 release notes.
Question: I am working with a large multi-page TIFF file that is JPEG-compressed, and I need to remove a single page from it. I am using the tifffile Python package to process the TIFF, and I already know which page I want to remove based on metadata tags associated with that page. My current approach is to read all pages, modify the target page (either by skipping or replacing it), and write the rest back to a new TIFF file. Here’s what I’ve tried so far: import tifffile with tifffile.TiffFile('file') as tif: for i, page in enumerate(tif.pages): if some condition with tags is true: # Skip the page to delete or replace with a dummy page image_data = page.asarray(memmap=True) # Memory-mapped access to the page's data # Write the page to the output file writer.write( image_data, compression='jpeg', photometric=page.photometric, metadata=page.tags, )  However, this approach has several issues:  Memory Usage: Processing a large file consumes almost all available memory (I have 32GB of RAM, but it uses up to 28GB), which makes it unfeasible for large files.  Compression Issues: Different compression methods like LZW, ZSTD, and JPEG create files of vastly different sizes, and some are much larger than the original.  Performance: Using methods like strips or chunking leads to very slow processing, taking too long to delete a single page.  Output file size: The size of the output file with using a different compression method makes it too big! (3GB Input on JPEG to 50GB+ output on LZW)   Is there any way in Python to efficiently remove a single page from a large multi-page TIFF file without consuming too much memory or taking forever? I’ve seen some .NET packages that can delete a page in-place—does Python have a similar solution?

Answer: I've created a Python package to handle it. While it can be made more extensible, it efficiently solves the problem without loading all the image data into memory. Core Idea: The package works by:  Reconstructing the IFD (Image File Directory) chain: It removes the IFD of concern while keeping references to the original image data. Adjusting Metadata: The metadata pointing to the "label" information is updated implicitly during the reconstruction process. Memory Efficiency: By referencing actual image data, the package avoids the need to load large image files into memory.  Installation: You can install the package directly from PyPI: pip install tiff-wsi-label-removal  Usage: Once installed, you can use the remove-label command-line tool to remove labels from a TIFF file: remove-label <input_tiff_file> <output_tiff_file>  Current Limitations and Future Work: The package is functional, but there’s room for improvement, including making it more extensible. Suggestions from the comments here and additional planned features are being tracked in the description of the package's PyPI page. Any feedback is welcome!
Question: I want to create an .mlpackage or .mlmodel file which I can import in Xcode to do image segmentation. For this, I want to use the segmentation package within YOLO to check out if it fit my needs. The problem now is that this script creates an .mlpackage file which only accepts images with a fixed size (640x640): from ultralytics import YOLO model = YOLO("yolo11n-seg.pt") model.export(format="coreml")  I want the change something here, probably with coremltools, to handle unbounded ranges (I want to handle arbitrary sized images). It's described a bit here: https://apple.github.io/coremltools/docs-guides/source/flexible-inputs.html#enable-unbounded-ranges, but I don't understand how I can implement it with my script.

Answer: How to export YOLO segmentation model with flexible input sizes from ultralytics import YOLO import coremltools as ct # Export to torchscript first model = YOLO("yolov8n-seg.pt") model.export(format="torchscript") # Convert to CoreML with flexible input size input_shape = ct.Shape( shape=(1, 3, ct.RangeDim(lower_bound=32, upper_bound=1024, default=640), ct.RangeDim(lower_bound=32, upper_bound=1024, default=640)) ) mlmodel = ct.convert( "yolov8n-seg.torchscript", inputs=[ct.ImageType( name="images", shape=input_shape, color_layout=ct.colorlayout.RGB, scale=1.0/255.0 )], minimum_deployment_target=ct.target.iOS16 ) mlmodel.save("yolov8n-seg-flexible.mlpackage")  This creates an .mlpackage that accepts images from 32 32 to 1024 1024 (you can modify these bounds as needed). Default is 640 640. Read about the stuff here:  https://apple.github.io/coremltools/docs-guides/source/flexible-inputs.html  https://docs.ultralytics.com/modes/export/  https://apple.github.io/coremltools/source/coremltools.converters.mil.input_types.html#coremltools.converters.mil.input_types.ImageType
Question: My question is very simple, but I can't find how to do it in the Snakemake documentation. Let's say I have a very long string to expand, like : rule all: input: expand("sim_files/test_nGen{ngen}_N{N}_S{S}_NR{NR}_DG{DG}_SS{SS}/test_nGen{ngen}_N{N}_S{S}_NR{NR}_DG{DG}_SS{SS}.yaml", ngen=list_ngen, N=list_N, S=list_S, NR=list_NR, DG=list_DG, SS=list_SS)  I want to save my long string in a variable, like this: prefix = "test_nGen{ngen}_N{N}_S{S}_NR{NR}_DG{DG}_SS{SS}" rule all: input: expand("sim_files/{prefix}/{prefix}.yaml",ngen=list_ngen, N=list_N, S=list_S, NR=list_NR, DG=list_DG, SS=list_SS)  But when I do that, Snakemake doesn't expand inside the wildcard 'prefix': WildcardError in file /home/bunelpau/Travail/DossierSync/test_snt/Snakefile, line 13: No values given for wildcard 'prefix'.  Do I have to use this super long string, or is there a way to save it in some variable like I want to?

Answer: Use an f-string to format the string before expansion: prefix = "test_nGen{ngen}_N{N}_S{S}_NR{NR}_DG{DG}_SS{SS}" rule all: input: expand(f"sim_files/{prefix}/{prefix}.yaml",ngen=list_ngen, N=list_N, S=list_S, NR=list_NR, DG=list_DG, SS=list_SS)