eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: When installing with pip, pip install sqlalchemy all is ok. When installing with poetry I am getting the error ➜ backend poetry add sqlalchemy Using version ^1.4.23 for SQLAlchemy Updating dependencies Resolving dependencies... (0.1s) AttributeError 'EmptyConstraint' object has no attribute 'allows' at ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/version/markers.py:291 in validate 287│ 288│ if self._name not in environment: 289│ return True 290│ → 291│ return self._constraint.allows(self._parser(environment[self._name])) 292│ 293│ def without_extras(self): # type: () -> MarkerTypes 294│ return self.exclude("extra") 295│ ➜ backend 

Answer: Try poetry self update, then poetry update.
Question: I have been tasked with creating a small application that allows the user to: browse and choose a specific folder that will contain the files to be checked daily, browse and choose a specific folder that will receive the copied files, and manually initiate the 'file check' process that is performed by the script. In order to do that, I need to get the last modified time and add an if statement that checks if the file is over 24hrs old, then it will move the file over to the correct folder. I think I know how to do all that. Where I'm running into issues is with the os.path.getmtime method. As far as i understand it, I cannot get the modified time of multiple files. I have to name the file specifically. So the '../File Transfer/Old Files/' doesnt work. I keep getting the same error: FileNotFoundError: [WinError 2] The system cannot find the file specified: 'LICENSE-MIT.txt'  (that's the first txt file in the Old Files folder) if I change the code to: x = os.path.getmtime(../File Transfer/Old Files/LICENSE-MIT.txt)  It works, but with one complication. It gives me the time in seconds which is in the 449 billion range. I'm able to convert that number to a usable number by using this code: y = int(x) z = y//60//60 print(z)  This keeps it as an int and not a float, and converts it from seconds to hours. Then I would just do a >=24 if statement to see if it's older than a day. HOWEVER, this is still just one the single file basis, which really doesnt work for what i want the program to do. I want it to do this with all files in a folder, regardless of what the names of those files are, or how many of them there are. ANY help would be super appreciated! Here's the script I'm using to iterate the folder and get each file mod time individually: for file in os.listdir('../File Transfer/Old Files/'): if file.endswith ('.txt'): time_mod = os.path.getmtime(file) print(time_mod)  Ignore the last line. That was just for me so see the results for troubleshooting.

Answer: The os.listdir() method lists the files of the given path excluding the path, hence you will need to concatenate the path yourself: for file in os.listdir('../File Transfer/Old Files/'): if file.endswith('.txt'): time_mod = os.path.getmtime('../File Transfer/Old Files/' + file) print(time_mod)  The glob.glob() method works great in cases like this: import os import glob for file in glob.globr('../File Transfer/Old Files/*.txt'): time_mod = os.path.getmtime('../File Transfer/Old Files/' + file) print(time_mod)  You can get the amount of hours passed since the last modification of each file like so: import os from time import time PATH = '../File Transfer/Old Files/' for file in os.listdir(PATH): if file.endswith('.txt'): time_mod = time() - os.path.getmtime(PATH + file) print(time_mod // 3600)
Question: I want to run run.sh to check the results of my codes, which takes about 30 hours. But I can't wait to add other features to my codes. However, I found there are some potential dangers:  Edit shell script while it's running Edit shell script and python script while it's running  In my case, I want run.sh running with all files are fixed while I keep editing more features on python files, which may have bugs. For example, here is my git log * commit 6d1931c0f5f6f7030f1a93d4f7496527c3711c1f (HEAD -> master) | Author: deeperlearner | Date: Fri Aug 13 10:37:21 2021 +0800 | | Version to be run by shell script |  How can I force run.sh run on commit 6d1931 while I keep doing commits? * commit 059706b752f4702bc5d0c830c87e812a7bbaae27 (HEAD -> master) | Author: deeperlearner | Date: Fri Aug 13 10:38:21 2021 +0800 | | Buggy commit that may destroy `run.sh` | * commit 6d1931c0f5f6f7030f1a93d4f7496527c3711c1f | Author: deeperlearner | Date: Fri Aug 13 10:37:21 2021 +0800 | | Version to be run by shell script |  I currently have solutions: use docker or just copy the whole codes to other directory. But I wonder if there is any method that can force shell script to execute on specific commit hash ID?

Answer: This is one use case for which git worktree is well suited. You basically create a new branch at the desired commit and make a copy of the working directory at that branch. For example, if you're at the top-level of your working tree and want to run the script on the current HEAD, just do: $ git worktree add ../bar Preparing worktree (new branch 'bar') HEAD is now at b2ceba9 Add d $ cd ../bar $ # Run long running script here $ cd ../foo $ # Edit files, create new commits, profit  (In this example, foo is the name of the top level directory of the repo.) There are options to worktree to work with a particular commit rather than HEAD. Read the fine manual for details.
Question: When I try to import the python 3 graphical library, turtle, on my m1 MacBook, I get an error message:  david@Davids-MacBook-Air Python Coding Files % /opt/homebrew/bin/python3 "/Users/david/Desktop/Python Coding Files/hello.py" Traceback (most recent call last): File "/Users/david/Desktop/Python Coding Files/hello.py", line 1, in <module> import turtle File "/opt/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/turtle.py", line 107, in <module> import tkinter as TK File "/opt/homebrew/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/tkinter/__init__.py", line 37, in <module> import _tkinter # If this fails your Python may not be configured for Tk ModuleNotFoundError: No module named '_tkinter'  I see it has the library tkinter written in it for some reason. This is weird because on my other Mac, an intel based Mac, with the same python 3 version, and no library such as tkinter install, I didnt get an error message when trying to import turtle Does anybody know what the problem is?

Answer: Python 3's turtle module has always used tkinter. Your other Mac must have tkinter installed. It used to be shipped with Python, but now you need to install it. Assuming you use HomeBrew: brew install python-tk
Question: Yesterday the following cell sequence in Google Colab would work.  (I am using colab-env to import environment variables from Google Drive.) This morning, when I run the same code, I get the following error.  It appears to be a new issue with s3fs and aiobotocore. I have some experience with Google Colab and library version dependency issues that I have previously solved by upgrading libraries in a particular order: !pip install --upgrade library_name  But I am a bit stuck this morning with this one. It is affecting all of my Google Colab notebooks so I thought that perhaps it is affecting others who are using data stored in Amazon AWS S3 with Google Colab. The version of s3fs that gets installed is 2021.07.0, which appears to be the latest. 

Answer: Indeed, the breakage was with the release of aiobotocore 1.4.0 (today, 20 Aug 2021), which is fixed in release 2021.08.0 of s3fs, also today.
Question: I'm trying to optimize the parameters learning rate and max_depth of a XGB regression model: from sklearn.model_selection import GridSearchCV from sklearn.model_selection import cross_val_score from xgboost import XGBRegressor param_grid = [ # trying learning rates from 0.01 to 0.2 {'eta ':[0.01, 0.05, 0.1, 0.2]}, # and max depth from 4 to 10 {'max_depth': [4, 6, 8, 10]} ] xgb_model = XGBRegressor(random_state = 0) grid_search = GridSearchCV(xgb_model, param_grid, cv=5, scoring='neg_root_mean_squared_error', return_train_score=True) grid_search.fit(final_OH_X_train_scaled, y_train)  final_OH_X_train_scaled is the training dataset that contains only numerical features. y_train is the training label - also numerical. This is returning the error: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan.  I've seen other similar questions, but couldn't find an answer yet. Also tried with: param_grid = [ # trying learning rates from 0.01 to 0.2 # and max depth from 4 to 10 {'eta ': [0.01, 0.05, 0.1, 0.2], 'max_depth': [4, 6, 8, 10]} ]  But it generates the same error. EDIT: Here's a sample of the data: final_OH_X_train_scaled.head()   y_train.head()   EDIT2: The data sample may be retrieved with: final_OH_X_train_scaled = pd.DataFrame([[0.540617 ,1.204666 ,1.670791 ,-0.445424 ,-0.890944 ,-0.491098 ,0.094999 ,1.522411 ,-0.247443 ,-0.559572 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,1.0 ,0.0 ,0.0], [0.117467 ,-2.351903 ,0.718969 ,-0.119721 ,-0.874705 ,-0.530832 ,-1.385230 ,2.126612 ,-0.947731 ,-0.156967 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,1.0 ,0.0 ,0.0 ,0.0 ,0.0], [0.901138 ,-0.208256 ,-0.019134 ,0.265250 ,-0.889128 ,-0.467753 ,0.169306 ,-0.973256 ,0.056164 ,-0.671978 , 0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,1.0 ,0.0 ,0.0], [2.074639 ,0.100602 ,-1.645121 ,0.929598 ,0.811911 ,1.364560 ,0.337242 ,0.435187 ,-0.388075 ,1.279959 , 0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,1.0], [2.198099 ,-0.496254 ,-0.917933 ,-1.418407 ,-0.975889 ,1.044495 ,0.254181 ,1.335285 ,2.079415 ,2.071974 , 0.0 ,0.0 ,0.0 ,0.0 ,0.0 ,1.0 ,0.0 ,0.0 ,0.0 ,0.0]], columns=['cont0' ,'cont1' ,'cont2' ,'cont3' ,'cont4' ,'cont5' ,'cont6' ,'cont7' ,'cont8' ,'cont9' ,'31' ,'32' ,'33' ,'34' ,'35' ,'36' ,'37' ,'38' ,'39' ,'40']) 

Answer: I was able to reproduce the problem and the code fails to fit because there is an extra space in your eta parameter! Instead of this: {'eta ':[0.01, 0.05, 0.1, 0.2]},...  Change it to this: {'eta':[0.01, 0.05, 0.1, 0.2]},...  The error message was unfortunately not very helpful.
Question: My dataframe is a long list of 4 letters, 'A', 'T', 'G','C', I need to count the frequency of each letter by index df = pd.DataFrame({'cases': ['ACCTTGTAGTGTATTTTATGACCAAATGACTTTTTCCCCCCAGTGGCTAATTTGTCTCAGGCCTGCGTCTTAAAGAGACACGGTAATGAGTAGGAAGTCCAGCGTGGTCTGGA','ACCTTGTACTGTATCTTATGACCAGATGACTTTTTCCACCCAGTGGCTAATTTGTCTCAGGCCTCCGTCTTAAAGAGACACGGTAATGAGTAGGAAGTCCAACGTGGTCTAGA','GCCTTGTACTGTATATTATGACCAAATGACTTTTTCCACCCATTGGCTAATTTGTCTCAGGCCTCCGTCTTAAAGAGACACGGAAATGAGTAGGAAGTCCAGCGTGGTCTAGA','ACCTTGTACTGTATATTATGACCAGATGACTTTTTCCACCCAGTGGCTAATTTGTCTCAGGCCTCCGTCTTAAAGAGACACGGTAATGAGTAGGAAGTCCAGCGTGGTCTAGA']})   cases 0 ACCTTGTAGTGTATTTTATGACCAAATGACTTTTTCCCCCCAGTGG... 1 ACCTTGTACTGTATCTTATGACCAGATGACTTTTTCCACCCAGTGG... 2 GCCTTGTACTGTATATTATGACCAAATGACTTTTTCCACCCATTGG... 3 ACCTTGTACTGTATATTATGACCAGATGACTTTTTCCACCCAGTGG... 4 ACCTTGTACTGTATATTATGACCAGATGACTTTTTCCACCCAGTGG... 5 ACCTTGTAGTGTATTTTATGACCAAATGACTTTTTCCCCCCAGTGG... 6 ACCTTGTACTGTATCTTATGACCAGATGACTTTTTCCACCCAGTGG... 7 GCCTTGTACTGTATATTATGACCAAATGACTTTTTCCACCCATTGG... 8 ACCTTGTACTGTATATTATGACCAGATGACTTTTTCCACCCAGTGG... 9 ACCTTGTACTGTATATTATGACCAGATGACTTTTTCCACCCAGTGG...  The result would be a new df of shape 4x113, i cannot figure out a pandas way to do this. Below is my non-pandas solution def freq_lists(dna_list): n = len(dna_list[0]) A = [0]*n T = [0]*n G = [0]*n C = [0]*n for dna in dna_list: for index, base in enumerate(dna): if base == 'A': A[index] += 1 elif base == 'C': C[index] += 1 elif base == 'G': G[index] += 1 elif base == 'T': T[index] += 1 return {'A': A, 'C': C, 'G': G, 'T': T} fdf = pd.DataFrame(freq_lists(df['cases'].to_list()))   A C G T 0 3 0 1 0 1 0 4 0 0 2 0 4 0 0 3 0 0 0 4 4 0 0 0 4 .. .. .. .. .. 108 0 4 0 0 109 0 0 0 4 110 3 0 1 0 111 0 0 4 0 112 4 0 0 0  To clarify the first row is obtained by summing up the counts of the first str in the case column which is AAGA -> A: 3, C:0, G:1 T:0

Answer: Use collections.Counter: from collections import Counter df['cases'].apply(lambda x: pd.Series(Counter(x)))  output:  A C T G 0 27 24 34 28 1 29 26 33 25 2 30 25 33 25 3 29 25 33 26  The other way around it not as sexy: pd.DataFrame([Counter(i) for i in list(zip(*df['cases'].apply(list).values))] ).fillna(0).astype(int)  or (df['cases'].apply(lambda x: pd.Series(list(x))) .apply(pd.value_counts) .T.fillna(0).astype(int) )  output:  A G C T 0 3 1 0 0 1 0 0 4 0 2 0 0 4 0 ... 111 0 4 0 0 112 4 0 0 0
Question: Pyinstaller recently added a splash screen option (yay!) but the splash stays open the entire time the exe is running. I need it because my file opens very slowly and I want to warn the user not to close the window. Is there a way I can get the splash screen to close when the gui opens?

Answer: from pyinstaller docs: import pyi_splash # Update the text on the splash screen pyi_splash.update_text("PyInstaller is a great software!") pyi_splash.update_text("Second time's a charm!") # Close the splash screen. It does not matter when the call # to this function is made, the splash screen remains open until # this function is called or the Python program is terminated. pyi_splash.close()
Question: I have created my own ERC-20 token (AJR) and deploy on Ethereum private node, Now I want to list all the transaction by Token name. Also, I need to list down all the token holders by using contract address or token name. I try to fetch using web3 but I get only symbol, name, total supply, etc. but not token holders or transactions Below is my sample code: from web3 import Web3 Web3 = Web3(Web3.HTTPProvider('http://127.0.0.1:8545')) contract_instance = Web3.eth.contract(contract_address, abi=abi) print(contract_instance.functions.name().call()) 

Answer: Token holders are not directly available through the RPC protocol and RPC wrappers such as Web3. Information about token holders is stored on the blockchain in the token contract (or some of its dependencies), usually in the form of a mapping. Which means that you can't just loop through all of the holders, but you need to know the address and ask for their balance. // the key is the holder address, the value is their token amount mapping (address => uint256) public balanceOf;  But - the ERC-20 standard defines the Transfer() event that the token contract should emit when a transfer occurs. mapping (address => uint256) public balanceOf; event Transfer(address indexed _from, address indexed _to, uint256 _amount); function transfer(address _to, uint256 _amount) external returns (bool) { balanceOf[msg.sender] -= _amount; balanceOf[_to] += _amount; emit Transfer(msg.sender, _to, _amount); return true; }  So you'll need to build and maintain a database of holders from all Transfer() event logs emitted by this token contract. Collect past event logs to build the historic data, and subscribe to newly emitted logs to keep it up-to-date. Then you can aggregate all of this raw transfer data to the form of "address => current balance" and filter only addresses that have non-zero balance in your searchable DB. Docs:  Get past event logs in Web3 - link Subscribe to new event logs in Web3 - link  The same way is actually used by blockchain explorers. They scan each transaction for Transfer() events and if the emitter is a token contract, they update the token balances in their separate DB. The list of all holders (from this separate DB) is then displayed on the token detail page.
Question: I am drawing a Confusion Matrix in fastai with following code: interp = ClassificationInterpretation.from_learner(learn) interp.plot_confusion_matrix()  But I end up with a super small matrix because I have around 20 categories:  I have found the related question for sklearns but don't know how to apply it to fastai (because we don't use pyplot directly.

Answer: If you check the code of the function ClassificationInterpretation.plot_confusion_matrix (in file fastai / interpret.py), this is what you see:  def plot_confusion_matrix(self, normalize=False, title='Confusion matrix', cmap="Blues", norm_dec=2, plot_txt=True, **kwargs): "Plot the confusion matrix, with `title` and using `cmap`." # This function is mainly copied from the sklearn docs cm = self.confusion_matrix() if normalize: cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis] fig = plt.figure(**kwargs) plt.imshow(cm, interpolation='nearest', cmap=cmap) plt.title(title) tick_marks = np.arange(len(self.vocab)) plt.xticks(tick_marks, self.vocab, rotation=90) plt.yticks(tick_marks, self.vocab, rotation=0)  The key here is the line fig = plt.figure(**kwargs), so basically, the function plot_confusion_matrix will propagate its parameters to the plot function. So you could use either one of these:  dpi=xxx (e.g. dpi=200) figsize=(xxx, yyy)  See this post on StackOverflow about the relations they have with each other: https://stackoverflow.com/a/47639545/1603480 So in your case, you could just do: interp.plot_confusion_matrix(figsize=(12, 12))  And the Confusion Matrix would look like:  FYI: this also applies to other plot functions, like interp.plot_top_losses(20, nrows=5, figsize=(25, 25))
Question: I am running a sentence transformer model and trying to truncate my tokens, but it doesn't appear to be working. My code is from transformers import AutoModel, AutoTokenizer model_name = "sentence-transformers/paraphrase-MiniLM-L6-v2" model = AutoModel.from_pretrained(model_name) tokenizer = AutoTokenizer.from_pretrained(model_name) text_tokens = tokenizer(text, padding=True, truncation=True, return_tensors="pt") text_embedding = model(**text_tokens)["pooler_output"]  I keep getting the following warning: Token indices sequence length is longer than the specified maximum sequence length for this model (909 > 512). Running this sequence through the model will result in indexing errors  I am wondering why setting truncation=True is not truncating my text to the desired length?

Answer: You need to add the max_length parameter while creating the tokenizer like below: text_tokens = tokenizer(text, padding=True, max_length=512, truncation=True, return_tensors="pt") Reason: truncation=True without max_length parameter takes sequence length equal to maximum acceptable input length by the model. It is 1e30 or 1000000000000000019884624838656 for this model. You can check by printing out tokenizer.model_max_length. According to the Huggingface documentation about truncation,  True or 'only_first' truncate to a maximum length specified by the max_length argument or the maximum length accepted by the model if no max_length is provided (max_length=None).
Question: I have two lists: nums = [2, 3, 5] mylist = ["aaa", "bbb", "ccc"]  What I am trying to achieve is repeat each element in mylist as many times as the corresponding values in nums. Expected output is: ["aaa", "aaa", "bbb", "bbb", "bbb", "ccc", "ccc", "ccc", "ccc", "ccc"] 

Answer: Let us do repeat import pandas as pd pd.Series(mylist).repeat(nums).tolist() Out[554]: ['aaa', 'aaa', 'bbb', 'bbb', 'bbb', 'ccc', 'ccc', 'ccc', 'ccc', 'ccc']  Or try numpy import numpy as np np.repeat(mylist,nums) Out[556]: array(['aaa', 'aaa', 'bbb', 'bbb', 'bbb', 'ccc', 'ccc', 'ccc', 'ccc', 'ccc'], dtype='<U3')
Question: This is a dummy sample of my dataframe data = [ [3273, "city y", [["ids", 27], ["smf", 13], ["tlk", 35], ["thr", 24]]], [3213, "city x", [["smf", 23], ["tlk", 15], ["ids", 17], ["thr", 34]]], ] df = spark.createDataFrame( data, "city_id:long, city_name:string, cel:array<struct<carr:string, subs:int>>" ) df.show(2, False) +-------+---------+--------------------------------------------+ |city_id|city_name|cel | +-------+---------+--------------------------------------------+ |3273 |city y |[[ids, 27], [smf, 13], [tlk, 35], [thr, 24]]| |3213 |city x |[[smf, 23], [tlk, 15], [ids, 17], [thr, 34]]| +-------+---------+--------------------------------------------+  I need to descendingly sort the array of column cel based on its subs value. It would be like this +-------+---------+--------------------------------------------+ |city_id|city_name|cel | +-------+---------+--------------------------------------------+ |3273 |city y |[[tlk, 35], [ids, 27], [thr, 24], [smf, 13]]| |3213 |city x |[[thr, 34], [smf, 23], [ids, 17], [tlk, 15]]| +-------+---------+--------------------------------------------+  Is there a way of doing it without using an UDF if possible? thanks I am using spark version 2.4.0

Answer: You can do it using some SQL lambda functions : df = df.withColumn( "cel", F.expr( "reverse(array_sort(transform(cel,x->struct(x['subs'] as subs,x['carr'] as carr))))" ), ) df.show() +-------+---------+--------------------------------------------+ |city_id|city_name|cel | +-------+---------+--------------------------------------------+ |3273 |city y |[[35, tlk], [27, ids], [24, thr], [13, smf]]| |3213 |city x |[[34, thr], [23, smf], [17, ids], [15, tlk]]| +-------+---------+--------------------------------------------+ df.printSchema() root |-- city_id: long (nullable = true) |-- city_name: string (nullable = true) |-- cel: array (nullable = true) | |-- element: struct (containsNull = false) | | |-- subs: integer (nullable = true) | | |-- carr: string (nullable = true)
Question: I am trying to run a python async app with an asyncioscheduler scheduled job but the APScheduler fails during build because of this error:  'Only timezones from the pytz library are supported' error  I do include pytz in my app and i am passing the timezone. What is causing the error? I am calling the asyncioscheduler in a class where i create job manager: from apscheduler.schedulers.asyncio import AsyncIOScheduler class ScheduleManager: def __init__(self) -> None: self.scheduler = AsyncIOScheduler() def start(self): self.scheduler.start() def stop(self): self.scheduler.shutdown() def add_seconds_interval_job(self, callback, interval : int): self.scheduler.add_job(callback, 'interval', seconds = interval) def add_minutes_interval_job(self, callback, interval : int): self.scheduler.add_job(callback, 'interval', minutes = interval) def add_hours_interval_job(self, callback, interval : int): self.scheduler.add_job(callback, 'interval', hours = interval) def add_days_interval_job(self, callback, interval : int): self.scheduler.add_job(callback, 'interval', days = interval)  then i call this manager from my application like : from jobs import ScheduleManager, ConfigJob class AppInitializer: def __init__(self) -> None: self.schedule_manager = ScheduleManager() self.config__job = ConfigJob() async def initialize(self, app, loop): self.schedule_manager.add_seconds_interval_job(self.config_job.run, 5) self.schedule_manager.start() 

Answer: The tzlocal library switched from pytz to zoneinfo timezones in 3.0 and APScheduler 3.x is not compatible with those. Due to this, APScheduler 3.7.0 has tzlocal pinned to v2.x. If you're getting tzlocal 3.0 installed through APScheduler, you're using an old version. Please upgrade.
Question: Database:     id trade token     1 abc 5523   2 fdfd 5145   3 sdfd 2899     Code: def db_fetchquery(sql): conn = psycopg2.connect(database="trade", user='postgres', password='jps', host='127.0.0.1', port= '5432') cursor = conn.cursor() conn.autocommit = True cursor.execute(sql) row = cursor.rowcount if row >= 1: data = cursor.fetchall() conn.close() return data conn.close() return False print(db_fetchquery("SELECT token FROM script"))  Result:  [(5523,),(5145,),(2899,)]  But I need results as:  [5523,5145,2899]  I also tried print(db_fetchquery("SELECT zerodha FROM script")[0]) but this gave result as:- [(5523,)] Also, why is there ',' / list inside list when I am fetching only one column?

Answer: Not sure if you are able to do that without further processing but I would do it like this: data = [x[0] for x in data]  which convert the list of tuples to a 1D list