eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to iterate through a folder and delete any file that is a duplicate image (but different name). After running this script all files get deleted except for one. There are at least a dozen unique ones out of about 5,000. Any help understanding why this is happening would be appreciated. import os import cv2 directory = r'C:\Users\Grid\scratch' for filename in os.listdir(directory): a=directory+'\\'+filename n=(cv2.imread(a)) q=0 for filename in os.listdir(directory): b=directory+'\\'+filename m=(cv2.imread(b)) comparison = n == m equal_arrays = comparison.all() if equal_arrays==True and q==1: os.remove(b) q=1 

Answer: There are a number of issues with your code, but I am going to suggest an alternate strategy. A more efficient way would be to collect md5 or sha1 hashes of the files, into a set or some other container, while iterating the directory. Then when you calculate the hashes you can check if that particular hash already exists in the collection which would indicate the file is a duplicate of one that you have already seen and therefore should be removed. Additional pointers: You only need to get the directory contents once. So there should be no reason to call os.listdir() more than once on the same directory. for example: import hashlib import os hashes = set() for filename in os.listdir(directory): path = os.path.join(directory, filename) digest = hashlib.sha1(open(path,'rb').read()).digest() if digest not in hashes: hashes.add(digest) else: os.remove(path)  You can use a more secure hash if you would like but the chances of encountering a collision are astronomically low.
Question: @overload def setSize(self,size:tuple[int|str])->None: ''' Set image size (width,height) ''' try:self.options.append(f"width=\"{str(size[0])}\" height=\"{str(size[1])}\"") except IndexError:print("Error reading the size, aborting") @overload def setSize(self,width:int|str,height:int|str)->None: ''' Set image Size ''' self.setSize((width,height))  This is my code and I called this function as var.setSize((500,500)) which would normally call the top one but I got this error: NotImplementedError: You should not call an overloaded function. A series of @overload-decorated functions outside a stub module should always be followed by an implementation that is not @overload-ed. 

Answer: When writing code with function overloading in Python, it is important to remember that  [y]ou should not call an overloaded function. A series of @overload-decorated functions outside a stub module should always be followed by an implementation that is not @overload-ed.  This is because @overload-ed functions are intended to simply declare the types of the arguments without providing an actual implementation. Therefore, an implementation must be provided following the @overload-ed functions in order for the code to be valid.
Question: minimal example:  from statsforecast import StatsForecast from statsforecast.models import AutoARIMA import pandas as pd df = pd.read_csv('https://datasets-nixtla.s3.amazonaws.com/air-passengers.csv') sf = StatsForecast( models = [AutoARIMA(season_length = 12)], freq = 'M', n_jobs=-1, verbose=True ) sf.fit(df)  How to get the parameters of the fitted model ? I know this is possible using pmdarima package, but pmdarima is way too slow and runs out of memory on large data. statsforecast seems promising, but only if there is a way to get the params

Answer: From this answer, the solution would be: sf.fitted_[0][0].model_['arma']  which will output a tuple of 7 values. I don't know the exact mapping of parameters to tuple values, but from this line it appears to be: (p, d, q, P, D, Q, constant)
Question:  File "f:\drug-traceability-blockchain-maddy\src\app.py", line 2, in <module> from web3 import Web3,HTTPProvider File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\web3\__init__.py", line 6, in <module> from eth_account import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_account\__init__.py", line 1, in <module> from eth_account.account import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_account\account.py", line 59, in <module> from eth_account.messages import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_account\messages.py", line 26, in <module> from eth_account._utils.structured_data.hashing import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_account\_utils\structured_data\hashing.py", line 9, in <module> from eth_abi import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_abi\__init__.py", line 6, in <module> from eth_abi.abi import ( # NOQA File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_abi\abi.py", line 1, in <module> from eth_abi.codec import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_abi\codec.py", line 16, in <module> from eth_abi.decoding import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_abi\decoding.py", line 14, in <module> from eth_abi.base import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_abi\base.py", line 7, in <module> from .grammar import ( File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\eth_abi\grammar.py", line 4, in <module> import parsimonious File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\parsimonious\__init__.py", line 9, in <module> from parsimonious.grammar import Grammar, TokenGrammar File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\parsimonious\grammar.py", line 14, in <module> from parsimonious.expressions import (Literal, Regex, Sequence, OneOf, File "C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\site-packages\parsimonious\expressions.py", line 9, in <module> from inspect import getargspec ImportError: cannot import name 'getargspec' from 'inspect' (C:\Users\Swapn\AppData\Local\Programs\Python\Python311\Lib\inspect.py)  please help me .. how to solve this error. this is blockchain project wgich i downloaded from github. after that i was npm install npm start truffle complile truffle migrate and for homepage of project run app.py then i was receive this error.Image of running errors

Answer: Try uninstalling web3 using pip uninstall web3.py and install the latest version from github using pip install git+https://github.com/ethereum/web3.py.git
Question: In FastAPI I had the following function that I used to open and close a DB session: def get_db(): try: db = SessionLocal() yield db finally: db.close()  And within the routes of my API I would do something like that: @router.get("/") async def read_all_events(user: dict = Depends(get_current_user), db: Session = Depends(get_db)): logger.info("API read_all_events") if user is None: raise http_user_credentials_not_valid_exception() return db.query(models.Events).all()  You can see that I am injectin the session in the api call. So now i want to do something similar within a python function: def do_something(): #get person data from database #play with person data #save new person data in database #get cars data from database  So i am wondering if I should use the same approach than in FastAPI (i do not know how) or if i just should be openning and clossing the connection manually like that: def do_something(): try: db = SessionLocal() yield db #get person data from database #play with person data #save new person data in database #get cars data from database finally: db.close()  Thanks

Answer: The usage of yield in this case is so that Depends(get_db) returns the db session instance, so that it can be used in the fastapi route, and as soon as the fastapi route returns response to user, the finally clause (db.close()) will be executed. This is good because every request will be using a separate db session, and db connections will be closed after every route response. If you want to use the db session normally in a function, just get the db instance using db = SessionLocal(), and proceed to use the db instance in the function. Example: def do_something(): db = SessionLocal() event = db.query(models.Events).first() db.delete(event) db.commit() db.close()
Question: tqdm is a nice python library to keep track of progress through an iterable. It's default mode of operation is to repeatedly clear a line and redraw with a carriage but this produced quite nasty output when combined with logging. Is there a way I can get this to write to log files periodically rather than using this print? This is the best I've got is my own hacky implementation: def my_tqdm(iterable): "Like tqdm but us logging. Include estimated time and time taken." start = time.time() for i, item in enumerate(iterable): elapsed = time.time() - start rate = elapsed / (i + 1) estimated = rate * len(iterable) - elapsed num_items = len(iterable) LOGGER.info( "Processed %d of %d items (%.1f%%) in %.1fs (%.1fs remaining, %.1f s/item)", i, num_items, i / num_items * 100, elapsed, estimated, rate, ) yield item  But it'd be better if I could do this with tqdm itself so that people don't moan at me in code reviews.

Answer: You could redirect the outputs of the TQDM progress bar to a null device (e.g. /dev/null), and manually print/log the status bar whenever you want - either on every iteration, or at a certain interval. For example: import os import time import logging from tqdm import tqdm LOG_INTERVAL = 5 logging.basicConfig(level=logging.INFO) logger = logging.getLogger('tqdm_logger') progress_bar = tqdm(range(20), file=open(os.devnull, 'w')) for i in progress_bar: # do something meaningful instead... time.sleep(0.1) if progress_bar.n % LOG_INTERVAL == 0: logger.info(str(progress_bar))  This code block will produce the following outputs: INFO:tqdm_logger: 0%| | 0/20 [00:00<?, ?it/s] INFO:tqdm_logger: 25%|██▌ | 5/20 [00:00<00:01, 9.59it/s] INFO:tqdm_logger: 50%|█████ | 10/20 [00:01<00:01, 9.54it/s] INFO:tqdm_logger: 75%|███████▌ | 15/20 [00:01<00:00, 9.54it/s]
Question: I generated several datasets, and using classifiers, I predicted the distribution of clusters. I need to draw boundaries between clusters on the chart. In the form of lines or in the form of filled areas - it does not matter. Please let me know if there is any way to do this. My code: import numpy as np import matplotlib.pyplot as plt from sklearn.neighbors import KNeighborsClassifier from sklearn.datasets import make_moons, make_circles from sklearn.model_selection import train_test_split n_sample = 2000 def make_square(n_sample): data=np.array([0,[]]) data[0] = np.random.sample((n_sample,2)) for i in range(n_sample): if data[0][i][0] > 0.5 and data[0][i][1] > 0.5 or data[0][i][0] < 0.5 and data[0][i][1] < 0.5: data[1].append(1) else: data[1].append(0) return data datasets = [ make_circles(n_samples=n_sample, noise=0.09, factor=0.5), make_square(n_sample), make_moons(n_samples=n_sample, noise=0.12), ] ks=[] for data in datasets: X,y = data[0],data[1] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=33) classifier = KNeighborsClassifier(n_neighbors=1) classifier.fit(X_train, y_train) y_pred = classifier.predict(X_test) acc = classifier.score(X_test, y_test) accs = [] for i in range(1, 8): knn = KNeighborsClassifier(n_neighbors=i) knn.fit(X_train, y_train) pred_i = knn.predict(X_test) acc0 = knn.score(X_test, y_test) accs.append(acc0) plt.figure(figsize=(12, 6)) plt.plot(range(1, 8), accs, color='red', linestyle='dashed', marker='o', markerfacecolor='blue', markersize=10) plt.title('accs Score K Value') plt.xlabel('K Value') plt.ylabel('accs Score') print("Max Score:", max(accs), "k=",accs.index(max(accs))+1) ks.append(accs.index(max(accs))+1) for i in range(3): data = datasets[i] k = ks[i] X,y = data[0],data[1] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=33) classifier = KNeighborsClassifier(n_neighbors=k) classifier.fit(X_train, y_train) y_pred = classifier.predict(X_test) plt.figure(figsize=(9,9)) plt.title("Test") plt.scatter(X_test[:,0], X_test[:,1], c=y_test) plt.figure(figsize=(9,9)) plt.title("Predict") plt.scatter(X_test[:,0], X_test[:,1], c=y_pred)  Example output: enter image description here enter image description here

Answer: scikit-learn 1.1 introduced the DecisionBoundaryDisplay to assist with this sort of task. Following the use of make_moons and the KNeighborsClassifier in the question, we can fit the classifier on the dataset, invoke the DecisionBoundaryDisplay.from_estimator() method, then scatter the X data on the returned axis: import matplotlib.pyplot as plt from sklearn.datasets import make_moons from sklearn.neighbors import KNeighborsClassifier from sklearn.inspection import DecisionBoundaryDisplay X, y = make_moons(noise=0.2) clf = KNeighborsClassifier().fit(X, y) disp = DecisionBoundaryDisplay.from_estimator(clf, X, response_method="predict", alpha=0.3) disp.ax_.scatter(X[:, 0], X[:, 1], c=y) plt.show()  Resulting in something like this:
Question: When I use a plugin that requires python, it can't find it and barfs. The places that seem to being searched are:  Using -version I see both: +python/dyn +python3/dyn  However :echo has("python3") returns 0. I'm not sure if this is compile time config, or runtime-configurable via .vimrc. I'm not a python developer, and the few times I've ventured into that world were in the middle of the python2/python3 mess that turned me off completely. I've played around enough to have configured pyenv it seems, and get ╰─$ which python /Users/benlieb/.pyenv/shims/python ╰─$ python --version Python 3.10.3  Can anyone help shed light on what to do to get python3 findable/usable in my vim? Update: Following @romainl's suggestion below I set in my .vimrc set pythonthreedll=/Users/benlieb/.pyenv/shims/python  But getting the following error: 

Answer: After some time, I found the following works, thought it was not a fun path of discovery. let &pythonthreedll = trim(system("pyenv which python"))
Question: I'm evaluating two different unsupervised ML algorithms, Isolation Forest and LSTM Autoencoder model, to identify anomalies in a large time series data. This dataset includes mostly categorical data such as Ip Adresses, cloud subscription Ids,tenant Ids, userAgents, and client Application Ids. When reading a tutorial on an implementation of a Tensorflow's Decision Tree (TF-DF) model, it mentions that the model handles non-label categorical values natively and  there is no need for preprocessing in the form of one-hot encoding, normalization or extra is_present feature.  Does anybody know how Tensorflow handles the categorical features behind the scenes (assuming they do some transformation into a numeric representation)?

Answer: Tl;dr: There is a natural way of using categorical features in decision trees/forests that requires no encoding. Tensorflow Decision Forests uses this and a number of standard transformations to handle categorical features. Tensorflow Decision Forest (TF-DF) constructs decision tree / decision forest models. A single decision tree recursively splits the dataset along its features. Splits along categorical features can naturally be performed through so-called in-set conditions. For instance, a tree can express a condition like userAgents \in \{“Mozilla/5.0”, “InternetExplorer/10.0”\}. Other types of conditions are also possible. Tensorflow Decision Forests (TF-DF) can construct in-set conditions if the dataset contains categorical features. More specifically, Tensorflow Decision Forests uses the C++ library Yggdrasil Decision Forests (YDF) under the hood for any advanced computations. YDF offers three different algorithms for finding a good categorical split of the data. For example, the Random algorithm will just try out many possible splits at random and pick the best one. For performance and quality reasons, YDF also preprocesses categorical features: If a categorical value is very rare, YDF may consider it “out-of-dictionary”, the threshold for “rare” being user-configurable. Furthermore, YDF maps the categorical features to integers by decreasing item frequency, with the mapping stored as part of the model. Note that this is purely an internal encoding; the algorithms are aware that a feature is categorical, hence typical issues with integer encodings do not apply. Finally, Tensorflow Decision Forests (TF-DF) uses Keras, which expects classification tasks to have an integer label. Therefore, TF-DF users have to encode the label themselves or use the built-in pd_dataframe_to_tf_dataset. Note that this answer only applies to Tensorflow Decision Forests. Other parts of Tensorflow may need manual encoding.
Question: Running from imblearn.over_sampling import SMOTE will raise following error. --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) d:\A\OneDrive - UBC\ENGR\518 Machine Learning\Project\codes\model_training_laptop - Copy.ipynb Cell 2 in <cell line: 1>() ----> 1 from imblearn.over_sampling import SMOTE File e:\Anaconda\lib\site-packages\imblearn\__init__.py:52, in <module> 48 sys.stderr.write("Partial import of imblearn during the build process.\n") 49 # We are not importing the rest of scikit-learn during the build 50 # process, as it may not be compiled yet 51 else: ---> 52 from . import combine 53 from . import ensemble 54 from . import exceptions File e:\Anaconda\lib\site-packages\imblearn\combine\__init__.py:5, in <module> 1 """The :mod:`imblearn.combine` provides methods which combine 2 over-sampling and under-sampling. 3 """ ----> 5 from ._smote_enn import SMOTEENN 6 from ._smote_tomek import SMOTETomek 8 __all__ = ["SMOTEENN", "SMOTETomek"] File e:\Anaconda\lib\site-packages\imblearn\combine\_smote_enn.py:10, in <module> 7 from sklearn.base import clone 8 from sklearn.utils import check_X_y ---> 10 from ..base import BaseSampler 11 from ..over_sampling import SMOTE 12 from ..over_sampling.base import BaseOverSampler File e:\Anaconda\lib\site-packages\imblearn\base.py:15, in <module> 12 from sklearn.preprocessing import label_binarize 13 from sklearn.utils.multiclass import check_classification_targets ---> 15 from .utils import check_sampling_strategy, check_target_type 16 from .utils._validation import ArraysTransformer 17 from .utils._validation import _deprecate_positional_args File e:\Anaconda\lib\site-packages\imblearn\utils\__init__.py:7, in <module> 1 """ 2 The :mod:`imblearn.utils` module includes various utilities. 3 """ 5 from ._docstring import Substitution ----> 7 from ._validation import check_neighbors_object 8 from ._validation import check_target_type 9 from ._validation import check_sampling_strategy File e:\Anaconda\lib\site-packages\imblearn\utils\_validation.py:15, in <module> 12 import numpy as np 14 from sklearn.base import clone ---> 15 from sklearn.neighbors._base import KNeighborsMixin 16 from sklearn.neighbors import NearestNeighbors 17 from sklearn.utils import column_or_1d File e:\Anaconda\lib\site-packages\sklearn\neighbors\__init__.py:6, in <module> 1 """ 2 The :mod:`sklearn.neighbors` module implements the k-nearest neighbors 3 algorithm. 4 """ ----> 6 from ._ball_tree import BallTree 7 from ._kd_tree import KDTree 8 from ._distance_metric import DistanceMetric File sklearn\neighbors\_ball_tree.pyx:1, in init sklearn.neighbors._ball_tree() AttributeError: module 'sklearn.metrics._dist_metrics' has no attribute 'DistanceMetric32' 

Answer: This is probably a case where upgrading scikit-learn and imbalanced-learn will resolve the problem. pip install --upgrade scikit-learn pip install --upgrade imbalanced-learn  Not all versions of scikit-learn and imbalanced-learn are compatible with one another. Version 0.10.0 should be compatible with scikit-learn>=1.0.0 (e.g. discussion here).
Question: I have a generator gen, with the following properties:  it's quite expensive to make it yield (more expensive than creating the generator) the elements take up a fair amount of memory sometimes all of the __next__ calls will throw an exception, but creating the generator doesn't tell you when that will happen  I didn't implement the generator myself. Is there a way to make the generator yield its first element (I will do this in a try/except), without having the generator subsequently start on the second element if I loop through it afterwards? I thought of creating some code like this: try: first = next(gen) except StopIterator: return None except Exception: print("Generator throws exception on a yield") # looping also over the first element which we yielded already for thing in (first, *gen): do_something_complicated(thing)  Solutions I can see which are not very nice:  Create generator, test first element, create a new generator, loop through the second one. Put the entire for loop in a try/except; not so nice because the exception thrown by the yield is very general and it would potentially catch other things. Yield first element, test it, then reform a new generator from the first element and the rest of gen (ideally without extracting all of gen's elements into a list, since this could take a lot of memory).  For 3, which seems like the best solution, a nearly-there example would be the example I gave above, but I believe that would just extract all the elements of gen into a tuple before we start iterating, which I would like to avoid.

Answer: I think I have what you are looking for using more_itertools library: import more_itertools if __name__ == "__main__": generator = range(100) peekable_generator = more_itertools.peekable(generator) print(f"peek {peekable_generator.peek()}") print(f"next {next(peekable_generator)}") print(f"next {next(peekable_generator)}") output: peek 0 next 0 next 1  See documentation here: https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.peekable If I'm not mistaken the ability to peek at the first item is the key thing you need.
Question: Given the xarray below, I would like to add 10 to all negative values (i.e, -5 becomes 5, -4 becomes 6 ... -1 becomes 9, all values remain unchanged). a = xr.DataArray(np.arange(25).reshape(5, 5)-5, dims=("x", "y"))  I tried:  a[a<0]=10+a[a<0], but it returns 2-dimensional boolean indexing is not supported. Several attempts with a.where, but it seems that the other argument can only replace the mapped values with a constant rather than with indexed values.  I also considered using numpy as suggested here, but my actual dataset is ~ 80 Gb and loaded with dask and using numpy crashes my Jupyter console. Is there any way to achieve this using only xarray? Update I updated the code using @SpaceBurger and this. However my initial example was using a DataArray whereas my true problem is using a Dataset: a = xr.DataArray(np.arange(25).reshape(5, 5)-5, dims=("x", "y")) a = a.to_dataset(name='variable')  Now, if I do this: a1 = a['variable'] a2 = 10+a1.copy() a['variable'] = dask.array.where(a['variable'] < 0, a2, a1)  I get this error: MissingDimensionsError: cannot set variable 'variable' with 2-dimensional data without explicit dimension names. Pass a tuple of (dims, data) instead.  Can anyone suggest a proper syntax?

Answer: xarray’s where method is the way to go here - you can provide any other argument which can be broadcast against the condition argument and the original array: a['variable'] = a['variable'].where( a['variable'] >= 0, (a['variable'] + 10), )  This will work fine with dask and will handle your coordinates seamlessly. Note that if you do try this with a dataset, all of the variables in the dataset will be broadcast against the condition and other. so if you have some data vars that don't include all these dimensions they'll end up being repeated and weird. generally I recommend doing math/operations on DataArrays or variables as I have it in my answer.