Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Unittesting DRF Serializer validators one by one
Tags: python,django,rest

Question: We have an example Serializer class we'd like to test:
from rest_framework import serializers
class MySerializer(serializers.Serializer):
    fieldA = serializers.CharField()
    fieldB = serializers.CharField()
    def validate_fieldA(self,AVal):
        if len(AVal) < 3:
            raise serializers.ValidationError("AField must be at least 3 characters long")
        return AVal

    def validate_fieldB(self,BVal):
        if not len(BVal) < 3:
            raise serializers.ValidationError("BField must be at least 3 characters long")
        return BVal

This is a greatly simplified scenario, but it should do.
We want to write unittests for this Serializer class. My friend argues we should test using the .is_valid() method, like so
class TestMySerializer(unittest.TestCase):
    def test_validate_fieldA_long_enough(self):
        ser = MySerializer(data={"fieldA":"I'm long enough","fieldB":"Whatever"}),
        self.assertTrue(ser.is_valid())
        self.assertEqual(ser.validated_data["fieldA"],"I'm long enough")

    def test_validate_fieldA_too_short(self):
        ser = MySerializer(data={"fieldA":"x","fieldB":"Whatever"})
        self.assertFalse(ser.is_valid())
    
    #similarly tests for fieldB
    ...

I argue that unittests are supposed to be atomic and test only one "thing" at a time. By using the .is_valid() method we run every validator in the class instead of just the one we want to test. This introduces an unwanted dependency, where tests for fieldA validators may fail if there's something wrong with fieldB validators. So instead I would write my tests like so:
class TestMySerializer(unittest.TestCase):
    def setUp(self):
        self.ser = MySerializer()

    def test_fieldA_validator_long_enough(self):
        self.assertEqual(self.ser.validate_fieldA("I'm long enough"),"I'm long enough")

    def test_fieldA_validator_too_short(self):
        with self.assertRaises(serializers.ValidationError) as catcher:
            self.ser.validate_fieldA('x')
        self.assertEqual(str(catcher.exception),"AField must be at least 3 characters long")
    #same for fieldB validators
    ...

Which approach is better and why?

Answer: You should not, in my opinion, specifically test the validation of your overly long chain.
It makes no sense because you are not the author of the code that performs this validation.
You must test your serializer and its various validation and invalidation cases.
The maximum length validator was written in Django Rest Framework and has already been tested.
The unit test that you write verifies the validation of specific input data.
So, in my opinion, you should call the method that is typically called for this purpose, which in your case is is_valid().
You could also call run_validators(), but that would only test the Django Rest Framework's functionality, which is unnecessary in your case (I specify that there are relevant cases to test a lib included, but not here for me)
Title: Storing H2o models/MOJO outside the file system
Tags: python,machine-learning,h2o

Question: I'm investigating the possibility of storing MOJOs in cloud storage blobs and/or a database.  I have proof-of-concept code working that saves the MOJO to a file then loads the file and stores to the target (and vice-versa for loading), but I'd like to know if there's any way to skip the file step?  I've looked into python's BytesIO, but since the h2o mojo APIs all require a file-path I don't think I can use it.

Answer: It's possible using the H2O's REST API. Have a look at model.download_mojo() for the reference which gets the model from the backend and then persists it using the _process_response() method. You can have a look at h2o.upload_mojo() for the uploading part.
Title: pygsheets does not modify Date Format
Tags: python,pygsheets

Question: I'm trying to convert a date in a Google Spreadsheet column from 3/3/2023 to Friday March 3, 2023 with pygsheets.
The following code:
client = pygsheets.authorize(service_account_file="credentials.json")
test_sheet = client.open(titles[0])
test_worksheets = test_sheet.worksheets()
active = test_worksheets[0]
model_cell = pygsheets.Cell("A1")
model_cell.set_text_format("fontSize",18)
model_cell.set_vertical_alignment(pygsheets.VerticalAlignment.MIDDLE)
model_cell.set_number_format(pygsheets.FormatType.DATE, 'dddd+ mmmm yyy')
pygsheets.DataRange('A2', 'A', worksheet=active).apply_format(model_cell)

successfully changes the fontSize and VerticalAlignment attributes but does not change the date format. What is wrong with the code?
UPDATE:
It seems there are two things at play here. First, my date format string isn't what I wanted and so it's possible that gsheets wasn't able to interpret it and just ignored. I'm skeptical that's the case but it's possible. The format string I need to use is 'dddd", "mmmm" "d", "yyyy'.
Second and more importantly, I noticed that after I move the date values from one column to another there is a single quote (or possibly a tick mark) at the start of the date string. I remove this quote and the format changes.
Seems like pygsheets is adding a tick mark at the beginning of non-numerical dates (for example 3/3/2023) i'm guessing to preserve the original formatting. But when you call the batch updater it doesn't remove the tick mark.
I'm not entirely sure how to get around this but at least I know what I need to get around now.

Answer: This almost seems like it's a bug but it's maybe more of a feature request or at least an update to the documentation.
What I didn't mention in my question was that before I tried to modify the format of the column, I moved the values from another column into column A. In the process, it seems, pygsheets adds a single quote or tick mark to the beginning of the date strings - presumably to maintain the original formatting.
Unfortunately this single quote seems to get in the way of gsheets modifying the string into a Date format. When I remove the tick mark/single quote, the formatting started to show up. So, for my issue I just modified the Date format before moving the values.
Title: partial derivatives of Scipy smooth spline not working
Tags: python,scipy,spline

Question: I tried to take the second x derivative of a fitted smooth spline in Scipy, like following:
spline = SmoothBivariateSpline(x,y,z,kx=3,ky=1)
splinedxx = spline.partial_derivative(2,0)

This give me following error:
File "C:\Users\xxxxx\AppData\Roaming\Python\Python39\site-packages\scipy\interpolate\_fitpack2.py", line 988, in partial_derivative
newc, ier = dfitpack.pardtc(tx, ty, c, kx, ky, dx, dy)
dfitpack.error: ((0 <= nuy) && (nux < ky)) failed for 7th argument nuy: pardtc:nuy=0

The same error is given when I try the first derivative. What did I do wrong? Or is it something caused by my data?

Answer: set ky=3 solves the problem. It seems like the order of y also need to be greater than the order of differentiation on x.
Title: Why do these different outlier methods fail to detect outliers?
Tags: python,pandas,group-by,outliers

Question: I am trying to find the outliers by group for my dataframe. I have two groups: Group1 and Group2, and I am trying to find the best way to implement an outlier method
data = {'Group1':['A', 'A', 'A', 'B', 'B', 'B','A','A','B','B','B','A','A','A','B','B','B','A','A','A','B','B','B','A','A','A','A','A','B','B'], 'Group2':['C', 'C', 'C', 'C', 'D', 'D','C','D','C','C','D', 'C', 'C', 'D', 'D','C', 'C','D','D','D', 'D','C','D','C','C', 'D','C','D','C','C'], 'Age':[20, 21, 19, 24, 11, 15, 18, 1, 17,23, 35,2000,22,24,24,18,17,19,21,22,20,25,18,24,17,19,16,18,25,23]} 
df = pd.DataFrame(data) 

groups = df.groupby(['Group1', 'Group2'])
means = groups.Age.transform('mean')
stds = groups.Age.transform('std')

df['Flag'] = ~df.Age.between(means-stds*3, means+stds*3)

def flag_outlier(x):
    lower_limit  = np.mean(x) - np.std(x) * 3 
    upper_limit = np.mean(x) + np.std(x) * 3
    return (x>upper_limit)| (x<lower_limit)

df['Flag2'] = df.groupby(['Group1', 'Group2'])['Age'].apply(flag_outlier)

df["Flag3"] = df.groupby(['Group1', 'Group2'])['Age'].transform(lambda x: (x - x.mean()).abs() > 3*x.std())

However, all 3 methods fail to detect obvious outliers - for example, when Age is 2000, none of these methods treat it as an outlier. Is there a reason for this? Or is it possible that my code for all three outlier detection models is incorrect?
I have a strong feeling I've made a foolish mistake somewhere but I'm not sure where, so any help would be appreciated, thanks!

Answer: Within its group, that age of 2000 just isn't over 3 standard deviations away from the group mean. The group mean is 239.666667 and the group standard deviation is 660.129722.
It might look like an obvious outlier to you, but you don't have enough data to label it an outlier by that standard.
Title: How to wait but not block thread when getting a resource in python?
Tags: python,block

Question: Suppose we need to create a database connection pool, the requirement is that when a client tries to get a connection, if all exisiting connections are busy, then need to wait for 30 second before giving up, hope some connections are released by other client. So the naive solution is
def get_connection():
    if all_conn_are_busy:
        time.sleep(30)
        try to get connection again
    else:
        return conn

But since time.sleep(30) will block the thread, if 2 clients trying to get connection at the same time, it will block for 60 seconds. So is there any way to noblock it but also wait for some time?

Answer: (A.) You have already lost the battle if you're in this "waiting" regime.
(B.) It sounds like you want to have a single thread which is exclusively
dedicated to acquiring that next connection.
And if it is already busy trying to acquire one,
then 2nd caller should promptly suffer fatal error.
To have dozens of threads all clamoring for next connection
from a terminally overloaded DB server sounds like a recipe for disaster.
Consider pre-allocating connections at some slow rate,
so there is always one available for the next requestor.
Apparently you have some connection pool, and a policy
for discarding idle connections. Re-evaluate how well suited
that policy is in light of your current use cases.
Title: ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory
Tags: python,installation,virtualenv

Question: I created a conda v-env with Python 3.8.10 and when I do pip install -r requirements_demo.txt, I get the following error.
Processing /home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
ERROR: Could not install packages due to an OSError: \[Errno 2\] No such file or directory: '/System/Volumes/Data/home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work'

I didn't know why it succeeded on my windows machine but on my m1 mac it just refuses to work. I have tried adding sudo, -H, --user, and all those yielded the same error above. I don't know what is going on now.

Answer: I had a very similar issue and found this which solved it for me:
pip list --format=freeze > requirements.txt
Title: Locating indices with element 1 and converting to a list in Python
Tags: python,list,numpy

Question: I have an array A. I want to identify all indices with element 1 and print as a list. But I am getting an error. I present the expected output.
import numpy as np

A=np.array([[1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0]])
A1=np.where(A[0]==1)
A1.tolist()
print(A1)

The error is
in <module>
    A1.tolist()

AttributeError: 'tuple' object has no attribute 'tolist'

The expected output is
[[0, 2, 3, 5]]

Answer: The array is at the zeroth index of the tuple, so do
[A1[0].tolist()] and you will have your expected output.
Title: Compare Latest CSV with all CSV in directory and remove the matching from the latest and write new rows in new file with python
Tags: python,pandas,csv,export-to-csv

Question: the code will not work properly e.g. when the name of files are something else.
for example when the file name is carre123.csv, it wont compare correctly. but when I changed the file name to test123.csv it works fine.
here is the code
import os
import pandas as pd

# Set the directory where the CSV files are stored
directory = '/PATH/csv-files'

# Get a list of all the CSV files in the directory
csv_files = [os.path.join(directory, f) for f in os.listdir(directory) if f.endswith('.csv')]
#print(csv_files)

# Sort the CSV files by modification time and select the last file as the latest file
latest_file = sorted(csv_files, key=os.path.getmtime)[-1]
#print(latest_file)

# Read the contents of the latest CSV file into a pandas DataFrame
latest_data = pd.read_csv(latest_file)
#print(latest_data)

# Iterate over all the previous CSV files
for csv_file in csv_files[:-1]:
    # Read the contents of the previous CSV file into a pandas DataFrame
    prev_data = pd.read_csv(csv_file)
    #print(prev_data)

    # Identify the rows in the latest CSV file that match the rows in the previous CSV file
    matches = latest_data.isin(prev_data.to_dict('list')).all(axis=1)
    print(matches)

    # Remove the matching rows from the latest CSV file
    latest_data = latest_data[~matches]

# Write the remaining rows in the latest CSV file to a new file
latest_data.to_csv('/NEWPATH/diff.csv', index=False)

when the file name is carre123.csv, it wont compare correctly. but when I changed the file name to test123.csv it works fine.

Answer: I think your code has a bug, which may be what is causing the problem. The for loop is over csv_files[:-1] which is not sorted by modification time, so depending on the file names this may cause the loop to include latest_file. Try storing the sorted list, sorted(csv_files, key=os.path.getmtime), then select the last one for latest_file and loop over the remaining files. Maybe there is something else wrong too, but based on the example you provided, this looks like the only issue I can see that is obviously a problem.
Title: How to display all resulting values in Jupiter notebook?
Tags: python,graphene-python

Question: I have the following code:
import numpy as np
import matplotlib.pyplot as plt

d0 = 0.3330630630630631
a0 = 0.15469469469469468

theta = 2
nmax=15

# lattice vectors sublattice 1
a1= np.array([3/2*a0,3**0.5/2*a0,0])
a2= np.array([3/2*a0,-3**0.5/2*a0,0])

# lattice vectors sublattice 2
b1 = np.array([ np.cos(theta) * a1[0] - np.sin(theta) * a1[1], np.sin(theta) * a1[0] + np.cos(theta) * a1[1], 0 ])
b2 = np.array([ np.cos(theta) * a2[0] - np.sin(theta) * a2[1], np.sin(theta) * a2[0] + np.cos(theta) * a2[1], 0 ])

## coordinates for the unrotated layer sublattice a&b
coords1a = np.array([i * a1 + j * a2 for i in range(-nmax-1, nmax+1) for j in range(-nmax-1, nmax+1)])
coords1b = np.array([i * a1 + j * a2 + [a0,0.,0.] for i in range(-nmax-1, nmax+1) for j in range(-nmax-1, nmax+1)])

## coordinates for the rotated layer sublattice a&b
coords2a = np.array([i * b1 + j * b2 + [0.,0.,d0]  for i in range(-nmax-1, nmax+1) for j in range(-nmax-1, nmax+1)])
coords2b = np.array([i * b1 + j * b2 + [np.cos(theta)*a0,np.sin(theta)*a0,d0] for i in range(-nmax-1, nmax+1) for j in range(-nmax-1, nmax+1)])

coords1 = np.concatenate((coords1a, coords1b))
coords2 = np.concatenate((coords2a, coords2b))

When I try to display my coords1 and coords2 lists, python give me a list than continue only 6 items with ", ... ," .
I want to see all the values.
How can I do that?

Answer: to quickly see all the elements try
print(*coords1)
and
print(*coords2)
Title: ModuleNotFoundError: No module named 'cvzone' in visual studio code
Tags: python,visual-studio-code,cvzone

Question: I was trying to import cvzone and cv2 library in vs code but it gives this error no matter what I do.
Tried to install this through cmd with pip and vs code (again, with pip) but nothing changes
python version 3.11.2
pip version 22.3.1

Answer: Well, it turns out making a virtual environment, installing wheel and only then installing cvzone solves this issue.
Title: How to include both ends of a pandas date_range()
Tags: python,pandas,datetime

Question: From a pair of dates, I would like to create a list of dates at monthly frequency, including the months of both dates indicated.
import pandas as pd
import datetime

# Option 1
pd.date_range(datetime(2022, 1, 13),datetime(2022, 4, 5), freq='M', inclusive='both')
# Option 2
pd.date_range("2022-01-13", "2022-04-05", freq='M', inclusive='both')

both return the list: DatetimeIndex(['2022-01-31', '2022-02-28', '2022-03-31'], dtype='datetime64[ns]', freq='M'). However, I am expecting the outcome with a list of dates (4 long) with one date for each month: [january, february, mars, april]
If now we run:
pd.date_range("2022-01-13", "2022-04-05", freq='M', inclusive='right')

we still obtain the same result as before. It looks like inclusive has no effect on the outcome.
Pandas version. 1.5.3

Answer: This is because the Month definition, if you use Day you see the difference.
When you count in months there is no effect.
For inclusive :
both : a <= x <= b  (in math convention : [a, b])
neither : a < x < b  (in math convention : ]a, b[)
right : a < x <= b  (in math convention : ]a, b])
left : a <= x < b  (in math convention : [a, b[)
You can't include beyond the limits
Title: Powershell: se variable in command
Tags: python,powershell

Question: I'm facing a basic and stupid problem on Windows PowerShell. I'm almost sure it has already been answered somewhere but I can't find something working for me.
I simply would like to use a variable inside a command in PowerShell:
$VENV_DIR='C:\venv\'
python -m venv $VENV_DIR
$VENV_DIR\Scripts\python.exe --version

I expect to see Python 3.10.8 as result.
But I have this error:
 PS C:\> $VENV_DIR\Scripts\python.exe --version
At line:1 char:10
+ $VENV_DIR\Scripts\python.exe --version
+          ~~~~~~~~~~~~~~~~~~~
Unexpected token '\Scripts\python.exe' in expression or statement.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErr
   orRecordException
    + FullyQualifiedErrorId : UnexpectedToken

I tried a lot of different combinations, but none of them work
  97 $VENV_DIR\Scripts\python.exe --version
  98 `$VENV_DIR\Scripts\python.exe --version
  99 $VENV_DIR\\Scripts\python.exe --version
 100 $VENV_DIR\Scripts\python.exe --version
 101 "$VENV_DIR"\Scripts\python.exe --version
 102 "$VENV_DIR\Scripts\python.exe" --version
 103 "$VENV_DIR\Scripts\python.exe --version"
 104 ("$VENV_DIR\Scripts\python.exe --version")
 105 $("$VENV_DIR\Scripts\python.exe --version")
 106 -$("$VENV_DIR\Scripts\python.exe --version")
 107 -$("$VENV_DIR")\Scripts\python.exe --version
 108 -$($VENV_DIR)\Scripts\python.exe --version
 109 $VENV_DIR\Scripts\python.exe --version
 110 (echo $VENV_DIR)\Scripts\python.exe --version
 111 echo $VENV_DIR\Scripts\python.exe --version
 112 $(echo $VENV_DIR)\Scripts\python.exe --version
 113 -$(echo $VENV_DIR)\Scripts\python.exe --version
 114 -$("echo $VENV_DIR")\Scripts\python.exe --version
 115 echo $VENV_DIR\Scripts\python.exe --version

Could you please help? Thanks

Answer: Have you tried putting an ampersand before the python command?
Title: Boto3 Exception when Creating s3 Bucket
Tags: python,amazon-web-services,amazon-s3,boto3

Question: I'm trying to create a bucket using:
s3_client = boto3.client('s3',
                         aws_access_key_id=api_id,
                         aws_secret_access_key=apikey,
                         region_name='us-east-1')

bucket_name = 'test1'

s3_client.create_bucket(Bucket=bucket_name)

I've seen this same code everywhere all over SO and Github and it's meant to work but I'm getting this exception:
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

Which from my research means I should specify the region (I did so in the client here), I also tried this line (using us-east-2 just to test a different region):
s3_client.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': 'us-east-2'})

But that still throws an exception:
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The us-east-2 location constraint is incompatible for the region specific endpoint this request was sent to.

I also tried this code:
s3_client = boto3.client('s3',
                         aws_access_key_id=api_id,
                         aws_secret_access_key=apikey)

bucket_name = 'test1'

s3_client.create_bucket(Bucket=bucket_name)

And my .aws config file specified:
[default]
region = us-east-1
output = json

As far as I can tell, one of these methods should have worked. If it were an auth issue, I'd have gotten that exception instead.
Can anyone point me in the right direction?

Answer: The error message displayed by the boto3 client is misleading, the error is actually due to the bucket name that you specified not being globally unique. Every bucket that is created has to have a unique name.
Try doing the same but use a longer, more complex bucket name which is unlikely to collide with any existing bucket names.
Title: Boto3 Exception when Creating s3 Bucket
Tags: python,amazon-web-services,amazon-s3,boto3

Question: I'm trying to create a bucket using:
s3_client = boto3.client('s3',
                         aws_access_key_id=api_id,
                         aws_secret_access_key=apikey,
                         region_name='us-east-1')

bucket_name = 'test1'

s3_client.create_bucket(Bucket=bucket_name)

I've seen this same code everywhere all over SO and Github and it's meant to work but I'm getting this exception:
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.

Which from my research means I should specify the region (I did so in the client here), I also tried this line (using us-east-2 just to test a different region):
s3_client.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': 'us-east-2'})

But that still throws an exception:
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The us-east-2 location constraint is incompatible for the region specific endpoint this request was sent to.

I also tried this code:
s3_client = boto3.client('s3',
                         aws_access_key_id=api_id,
                         aws_secret_access_key=apikey)

bucket_name = 'test1'

s3_client.create_bucket(Bucket=bucket_name)

And my .aws config file specified:
[default]
region = us-east-1
output = json

As far as I can tell, one of these methods should have worked. If it were an auth issue, I'd have gotten that exception instead.
Can anyone point me in the right direction?

Answer: It works as long as I don't try to write into us-east-1, any of the other regions work just fine.
Title: Thinking about the best way to merge two DataFrame
Tags: python,pandas,dataframe,data-cleaning

Question: I'm looking for a way to merge df. However, I don't know what would be the best way to do this.
first df - metro cities/population/teams




Metropolitan area
Population (2016 est.)[8]
NHL




Phoenix
4661537
Coyotes


Los Angeles
13310447
Kings Ducks


Toronto
5928040
Maple Leafs


Boston
4794447
Bruins


Edmonton
1321426
Oilers


New York City
20153634
Rangers Islanders Devils




Second df - team/wins/losses




team
w
L




Los Angeles Kings
46
28


Phoenix Coyotes
37
30


Toronto Maple Leafs
49
26


Boston Bruins
50
20


Edmonton Oilers
29
44


New York Islanders
34
37




I tried to merge across teams. However, I need to arrange this data so that it collides in the Merge. I don't know how I would do that without looking at it case by case.
Note: The data set is much larger and with more cities and teams.
I had a little trouble presenting the DF here, so I only put 6 rows and the main columns.

Answer: If you are trying to get the city part of a NHL team name, you could for example:
Make a hash map which contains all the possible city names; {"Toronto": "Toronto"},
split the NHL TEAM string and check if the hash map contains any part of the string. If it does that's the city name.
With the limited amount of possible city names that's not too bad.
But I'm not exactly sure what you are trying to accomplish, you should elaborate and simplify your question.
Title: How to keep the capture of the previous window after closing another one?
Tags: python,tkinter

Question: I am making an application in which different windows open, and naturally, when the window is opened, the functionality of the previous window is not available until the current one closes.
It would seem that everything is fine, but for example: we open three windows - window_1, window_2, window_3 - one by one from each window. We see that when window_3 is active, all others are inactive. Next, we close the active window, and we see that suddenly, along with window_2, window_1 is also active.
How to solve this problem?
I wrote a simple code to fully understand this task:
from tkinter import *

class Second_Child_Win():

    def __init__(self):

        self.root = Toplevel()
        self.root.geometry('200x200')
        self.root.grab_set()

    def run(self, first_child_root):

        Button(self.root, text='Quit', command=self.root.destroy).pack()
        first_child_root.wait_window(self.root)


class First_Child_Win():

    def __init__(self):

        self.root = Toplevel()
        self.root.geometry('200x200')
        self.root.grab_set()

    def run(self, main_root):

        Button(self.root, text='click', command=self.open_new_win).pack()
        main_root.wait_window(self.root)

    def open_new_win(self):

        A = Second_Child_Win()
        A.run(self.root)


class Main():

    def __init__(self):

        self.root = Tk()
        self.root.geometry('200x200')

    def run(self):

        Button(self.root, text='click', command=self.open_new_win).pack()
        self.root.mainloop()

    def open_new_win(self):

        A = First_Child_Win()
        A.run(self.root)


A = Main()
A.run()

Answer: In general, the decision is quite simple. Simply, after ending the session of the current window, I write grab_set again
Title: AttributeError: module 'IPython.utils.traitlets' has no attribute 'Unicode'
Tags: module,google-colaboratory,attributeerror,kaggle,python-traitlets

Question: I am running a .ipynb notebook on a Kaggle server.
At the first code cell, when importing modules, specifically cv2_imshow from google.patches as follows,
from google.colab.patches import cv2_imshow
I get this error:
/opt/conda/lib/python3.7/site-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  warn("IPython.utils.traitlets has moved to a top-level traitlets package.")
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_27/1840971195.py in <module>
     18 
     19 # Display images using OpenCV
---> 20 from google.colab.patches import cv2_imshow                                                      # Importing cv2_imshow from google.patches to display images
     21 
     22 # Ignore warnings

/opt/conda/lib/python3.7/site-packages/google/colab/__init__.py in <module>
     24 from google.colab import _tensorflow_magics
     25 from google.colab import auth
---> 26 from google.colab import data_table
     27 from google.colab import drive
     28 from google.colab import files

/opt/conda/lib/python3.7/site-packages/google/colab/data_table.py in <module>
    164 
    165 
--> 166 class _JavascriptModuleFormatter(_IPython.core.formatters.BaseFormatter):
    167   format_type = _traitlets.Unicode(_JAVASCRIPT_MODULE_MIME_TYPE)
    168   print_method = _traitlets.ObjectName('_repr_javascript_module_')

/opt/conda/lib/python3.7/site-packages/google/colab/data_table.py in _JavascriptModuleFormatter()
    165 
    166 class _JavascriptModuleFormatter(_IPython.core.formatters.BaseFormatter):
--> 167   format_type = _traitlets.Unicode(_JAVASCRIPT_MODULE_MIME_TYPE)
    168   print_method = _traitlets.ObjectName('_repr_javascript_module_')
    169 

AttributeError: module 'IPython.utils.traitlets' has no attribute 'Unicode'

After running
from traitlets import *
print(traitlets)
<module 'traitlets.traitlets' from '/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py'>

and re-running the problem line, to deal with the top part of the error message,
/opt/conda/lib/python3.7/site-packages/IPython/utils/traitlets.py:5: UserWarning: IPython.utils.traitlets has moved to a top-level traitlets package.
  warn("IPython.utils.traitlets has moved to a top-level traitlets package.")

This part of the error message dissappears but all else remains the same.
google-colab 1.0.0

Answer: The warning message is saying that traitlets is now a top level package, and not still under the IPython.utils package.
You can fix the error by editing the data_table.py file ( /opt/conda/lib/python3.7/site-packages/google/colab/data_table.py). Change the line importing the traitlets package from
from IPython.utils import traitlets as _traitlets
to
import traitlets as _traitlets
I have no idea why this error arose, but this fixed it for me anyway.
Title: Sending multiple transactions as once
Tags: python,solidity,web3py,ganache

Question: What are some ways to send multiple transactions at once to a ganache blockchain ?
I am using the web3.py library.

Answer: Ethereum does not support sending multiple transactions once.
Only way to do any kind of multi-transaction logic is to write your own smart contract.
Alternatively you can use another blockchain, like Cosmos SDK based ones, which support sending multiple transactions once.
Title: Import error when importing wx on Apple silicon
Tags: python-3.x,wxpython

Question: I have a Python 3.8 project that was created on a Intel Mac that I'm trying to get working on my Apple silicon M2 MacBook Pro.
wxPython installed fine using brew, but importing wx causes the following import error:
Python 3.11.2 (main, Feb 16 2023, 02:55:59) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/lib/python3.11/site-packages/wx/__init__.py", line 17, in <module>
    from wx.core import *
  File "/opt/homebrew/lib/python3.11/site-packages/wx/core.py", line 12, in <module>
    from ._core import *
ImportError: dlopen(/opt/homebrew/lib/python3.11/site-packages/wx/_core.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '__ZN10wxBoxSizer20InformFirstDirectionEiii'

I've tried re-installing wxPython clean, but didn't help.

Answer: Installing python3@3.10 solved my issue. I guess wxPython isn't working with 3.11. Thanks!
Title: Curl fails with SSL errors 56 and 35 when talking to a HTTPS Python web server
Tags: python,ssl,curl

Question: I have setup my own HTTPS server using Python:
from http.server import HTTPServer, BaseHTTPRequestHandler
import ssl

class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello, secure world!\n')

httpd = HTTPServer(('127.0.0.1', 4443), SimpleHTTPRequestHandler)
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain(
        certfile="/etc/letsencrypt/live/.../fullchain.pem",
        keyfile="/etc/letsencrypt/live/.../privkey.pem")
httpd.socket = context.wrap_socket(httpd.socket, server_side=True)
httpd.serve_forever()

For security purposes, my server runs in a VirtualBox VM with port forwarding from 443 to 4443. When I query the HTTPS server via curl locally on the VM, I get a response, but there is also an error:
$ curl -k 'https://locahost:4443'
Hello, secure world!
curl: (56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0

However, when I try to query it from the host, I only get an error:
$ curl -k 'https://127.0.0.1'
curl: (35) Unknown SSL protocol error in connection to 127.0.0.1:443

As you can see, in both calls I disable cert verification because I query my server by IP address and not the actual domain. The certs are for my personal domain and are signed by letsencrypt.com. When I tried using an actual domain on the host, I get the same error (34).
Why do I get errors in the curl call in the guest VM and why is it different when calling from the host?

Answer: Found bug in my code. HTTPServer needs to be constructed with ('0.0.0.0', 4443) to allow connections from other hosts.
Title: Getting data from streamlit instance
Tags: python,streamlit

Question: I made a streamlit app that locally writes to a CSV on the instance, and just realized now i have no way to get that data. I am pretty sure if I push up a change via github it'll overwrite the DB (can I use a gitignore maybe?)
Pretty dumb problem, open to any suggestions how to recover the data :)

Answer: To recover the data, you need to access the CSV file directly on the instance where the Streamlit app is running. You can try using a file transfer protocol (FTP) client to connect to the instance and download the CSV file. If you're on windows you'd use WinSCP for linux and Mac there is Filezilla.
For future purposes, it's better to use a database for storage and retrieval like MySQL or PostgresDB. You could also store the CSV in storage facility like S3 as a last resort. But Databases are the way to go.
Also yes you could add the CSV to your .gitignore file.
Title: Alembic raises ImportError cannot import name '_NONE_NAME'
Tags: python,sqlalchemy,alembic

Question: When I rebuilt a Docker image for my Python application that uses SQLAlchemy and Alembic, I started getting the following error when running migrations:
ImportError: cannot import name '_NONE_NAME' from 'sqlalchemy.sql.naming'

I didn't change my pinned requirements in requirements.txt. What is causing the issue?

Answer: I had this same problem since yesterday.
Reproducing the problem locally:
I noticed that creating a new venv and reinstalling the requirements reproduced the problem like in the container.
Diagnosis:
So, I created a new venv and reinstalled the requirements.txt.
Then i did pip freeze and compared the new installations to the pip freeze running on the old venv that was still working locally.
I found that alembic was installed witha  different versions.
My solution:
I downgraded alembic to be alembi==1.8.1 and now it works.
Hope this helps!
Title: Consuming messages from Kafka with different group IDs using confluent_kafka
Tags: python,apache-kafka,confluent-kafka-python

Question: I am using python and confluent_kafka
I am building a Queue Management for Kafka where we can view the pending(uncommitted) messages of each topic, delete topic and purge topic. I am facing the following problems.

I am using same group ID for all the consumers so that I can get the uncommitted messages.
I have 2 consumers one (say consumer1) consuming and committing and another one (say consumer2) just consuming without committing.

If I run consumer1 and consumer2 simultaneously only one consumer will start consuming and another just keep on waiting and hence cause heavy loading time in the frontend.
If I assign different group Id for each it works but, the messages committed by consumer1 are still readable by consumer2.
Example:
If I have pushed 100 messages and say consumer1 consumed 80 messages and when I try to consume from consumer2 it should consume only remaining 20, but it is consuming all 100 including the messages committed by consumer1.
How can I avoid that or solve?

Answer: Unclear what you mean by uncommitted. Any message in a topic has been committed by a producer.
From the consumer perspective, this isn't possible. Active Kafka consumers in the same group cannot be assigned the same partitions
More specifically, how would "consumer2" know when/if "consumer1" was "done consuming 80 records" without consumer1 becoming inactive?
If you have an idle consumer with only two consumers in the same group, sounds like you only have one partition... If you want both to be active at the same time, you'll need multiple partitions, but that won't help with any "visualizations" unless you persist your consumed data in some central location. At which point, Kafka Connect might be a better solution than Python.
If you want to view consumer lag (how far behind a consumer is processing), then there are other tools to do this, such as Burrow with its REST API. Otherwise, you need to use the get_watermark_offsets() function to find the topic's offsets and compare to the current polled record offset
Title: Importing playwright fails with "DLL load failed while importing _greenlet"
Tags: python,playwright-python

Question: Why does this error appear in the playwright file?
ImportError: DLL load failed while importing _greenlet: The specified module could not be found.

Here is my code:
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chrome.launch()
    page = browser.new_page()
    page.goto("https://www.youtube.com/watch?v=FK_5SQPq6nY&list=PLYDwWPRvXB8_W56h2C1z5zrlnAlvqpJ6A&index=1")
    page.screenshot(path="demo.png")
    browser.close()

Answer: The issue is from greenlet version, change greenlet version to 1.1.2. I have changed and it workes
Title: Access is denied when running a Python script from a command line in Windows 11
Tags: python,windows

Question: Suddenly during steps running python scripts I got an 'Access is denied' when running a script.
E:\crs\bde2\src>C:\python27\ArcGIS10.7\python.exe validate_csv.py aims
Access is denied.

Just trying to start python with no script or using -interactive produces no response.
However starting an IDE (pyscripter) and opening the file works as normal.
I spent two days following up this error message on web sites changing system settings, ACLs, permissions, developer overrides, admin permissions with no result.
Finally today I worked through yet another list of system settings and the last one suggested that I validate the system files (all fine) and then uninstall the app and reinstall.
[https://windowsreport.com/access-denied-windows-11/][1]

Answer: It turned out that the startup python.exe in the install directory c:\python27 had been set to zero bytes!
All I had to do was to restore the file (27k) and it is working again. Nothing to do with the windows system message putting me off the scent.
I do not know how the file was corrupted, but clearly it was.
Title: How do you stop the dag started with TriggerDagRunOperator?
Tags: python,airflow,directed-acyclic-graphs

Question: In my parent dag, I have a child dag started with the TriggerDagRunOperator. That child dag goes ahead and does its own thing.
my_trigger = TriggerDagRunOperator(
  task_id = "my_trigger",
  trigger_dag_id = "child_dag",
  wait_for_completion = True,
  execution_date="{{ logical_date }}",
)

However, somewhere down the line in my parent dag, something fails and I just want to stop everything. There's no point in the child dag still running if something failed in the parent dag.
I can set the state of the my_trigger task to State.FAILED and that will fail the task, but it won't stop the child dag from continuing to run until it completes.
So my question is how do you do stop the dag started by the TriggerDagRunOperator? I looked around members of my_trigger but I couldn't really find anything that would let me stop the dag. Does anyone have any ideas?

Answer: Launch the UI and find the DAG in the DAG List. Select each Task instance that is running or queued or scheduled, etc. and change the status to "failed". Then select the DAG instance and set the status to "failed".
Title: Use a pool of coroutine workers or one coroutine per task with Semaphore
Tags: python,python-asyncio

Question: Suppose I have one million web pages to download. Should I use a fixed number of coroutine workers to download them, or create a coroutine per url and use asyncio.Semaphore to limit the numbers of coroutines.
First option, fixed workers:
async def worker(queue):
    while True:
        try:
            url = queue.get_nowait()
        except QueueEmpty:
            break
        await download(url)

async def main():
    queue = asyncio.Queue()
    for url in urls:
        await queue.put(url)
    workers = [worker(queue) for _ in range(10)]
    await asyncio.wait(workers)

asyncio.run(main())

Second option, use semaphore:
async def worker(url, sema):
    async with sema:
        await download(url)

async def main():
    semaphore = asyncio.Semaphore(10)
    workers = [worker(url, semaphore) for url in urls]
    await asyncio.wait(workers)

asyncio.run(main())

Answer: The first approach (Queue + 10 tasks) won't equally distribute the workload (1M urls downloads) between fixed number of coroutines/tasks. When you'll add  simple ordinal indices to identify each worker during processing you may note that some workers will hold/occupy the queue for a longer time so some workers downloads a hundreds of urls and some - tens of thousands urls.
    While the 2nd approach (Semaphore) ensures that only a fixed number of urls will downloaded at a time by fixed number of active workers.
    The above should also count on a remote server max connections limit (in case if a large proportion of urls have same origin).
Title: python binning the Numbers in numpy
Tags: python,numpy

Question: I want to bin an array in the following way:
[0.1, 0.2, 0.3, ... 1.0].
I already try:
bins = np.array([0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9])
Digitalized_lables_train = np.digitize(lables_train, bins ) 

lables_train are numbers from 0-9.
But after this operation i still get bins between 1-10.

Answer: Got the answer:
just dividing my np.digitize(lables_train, bins ) with 10.
But for the case to bin in other float numbers i would like to know which other ways there could be.
Title: conda environments in team and shared Linux environment - best practices?
Tags: python,conda,sdlc,mamba

Question: Conda is great, but it assumes that people code completely independently much like the complete lack of namespace hierarchy in public python modules. Looking for best practices with 6 people all developing in largely the same conda environment. Let's pretend there won't be exceptions. No containers, which would make this easier.
I want to ask others what tips other have? Any other best practices anyone can recommend?
Here's what we are working on currently:

Having a "source" environment.yml and then having all developers work from an exported snapshot yml (unless they are developing the conda environment itself) so they are all developing in the exact same environment. you don't want surprises when someone's app goes to prod and you find out the environment it was developed under. This is just standard best practice in every language.

Deploying shared conda environments to QA and Prod. Conda environments take up several GB of space, which is a minor concern, but it also takes a lot of time to install. It's a lot of machinery to install for every application update. Yes, need to be concerned about versioning that conda environment, but it is a less a problem to have 2 or 3 versions in use at the same time. And yes, there are occasionally bugs in conda/mamba. It's a pain to work around them

Answer: Been using conda for a couple of years but lately abandoned it after it put my team into deadlock situations a couple of times.
First suggestion I'd give, if you want to remain in the conda system, is at least to switch to mamba, since conda at times can be exceedingly slow.
You did not specify what environment/languages you need to manage.
For my team it's 95% python, and we switched to poetry which we find a lot faster, and makes it easier to manage a consistent environment via its TOML config. We use it also to separate the productions project dependencies from the development env, where the latter is equipped with many linters/testers etc used in either local (eg pre-commit) or remote (eg github actions) workflows.
Of course if you'd need to install non python stuff that is not on PyPI, either conda or your platform's package managers (eg homebrew for Mac, apt for Debian Linux and derivatives) will be necessary.
Hope any of this is useful.
Title: Transformer encoder layer with pytorch : The shape of the 2D attn_mask is torch.Size([16, 512]), but should be (16, 16)
Tags: python,pytorch

Question: Here it's an minimal exemple of code :
encoder_layers = nn.TransformerEncoderLayer(512, 8,2048 ,0.5)
mask = torch.randint(0,2, (16,512)).bool()
text = torch.randn(16,512)
print(mask)
print(text)
encoder_layers(text,mask)


This gives me the following error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-b326564457ab> in <module>
      4 print(mask)
      5 print(text)
----> 6 encoder_layers(text,mask)

5 frames
/usr/local/lib/python3.9/dist-packages/torch/nn/functional.py in multi_head_attention_forward(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v, average_attn_weights)
   5067             correct_2d_size = (tgt_len, src_len)
   5068             if attn_mask.shape != correct_2d_size:
-> 5069                 raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.")
   5070             attn_mask = attn_mask.unsqueeze(0)
   5071         elif attn_mask.dim() == 3:

RuntimeError: The shape of the 2D attn_mask is torch.Size([16, 512]), but should be (16, 16).


I don't understand why that doesn't work because the mask length will be equal to the number of the tokens ?

Answer: Mask represents which query vectors can attend to which key vectors in the attention section. For example, in machine translation, the training batch has the entire sentence in the target language, but we don't want the queries at each word in the target language to attend to the keys for future words in that sentence. So at training time we would apply a mask to filter out the future keys for each query. Therefore the attention mask should be of shape [len(queries), len(keys)]. In your example, len(queries) = len(keys) = 16.
Title: How to upgrade pandas 2.0rc in VSCode
Tags: python,pandas,visual-studio-code

Question: Trying to upgrade to pandas 2.0rc in VSCode using the jupyter notebook extension.
! python -m pip install --upgrade pip
! pip install --pre pandas==2.0.0rc0
import pandas as pd

pd.__version__

The result is:
After the installation stuff...
Successfully installed pandas-2.0.0rc0

So why does this still appear for the version?
'1.5.2'

Answer: Easy answer. Restart VSCode. And then it works.
Title: Ufmt Format doesn't work on Py scripts with match case
Tags: python,format,match,case

Question: I know match case is a pretty recent addition to Python. ufmt (v2.0.1) still doesn't format files with match case statements in them. Any workarounds? Thanks
Here's the last thing I get from terminal when I run ufmt format on such a file
libcst._exceptions.ParserSyntaxError: Syntax Error @ 11:19.
Incomplete input. Encountered 'unit', but expected ';', or 'NEWLINE'.

            match unit:
                  ^

Tried running ufmt format on file with match case statement in them. Expected the file to be formatted accordingly. Got errors and the file wasn't formatted

Answer: This happen because µsort (the import sorter that µfmt uses) uses LibCST, which requires enabling its native Rust parser in order to support 3.10+ syntax, including match statements.
This is fixed in the latest version of µfmt (2.1.0) where it enables the LibCST native parser by default when formatting.
If you can't (or prefer not to) upgrade µfmt, then you can workaround this issue by setting LIBCST_PARSER_TYPE=native in your environment before running µfmt or µsort.