Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Concatenate video frames using AvgPooling
Tags: python,machine-learning,pytorch,computer-vision,data-science

Question: I have features extracted from 4 images. These images are video frames. And i want to combine them into one vector of shape (1 ,768) or (1, 512) Is AvgPooling the best way to do it?
import torch
input = torch.rand([1, 4, 768])
sumpool = torch.nn.AdaptiveAvgPool2d((1, 512))
sumpool(input).shape #torch.Size([1, 1, 512])

Also i tried MeanPooling:
result = torch.sum(visual_output, dim=1) / 4 #(1, 768)

But seems like i wrong somewhere. After using these combined features results are worse. Is everything correct?

Answer: Adaptive average pool adjusts sizes for pooling regions whereas mean pooling is similar to AvgPool2d, it solves by dividing the input feature map into several non-overlapping regions and computing the average of each region, assuming your input size is always different from output size created we get irregular results. Basic Pooling had this problem that is why Adaptive pooling came into being.
Title: Multi-Thread Binary Tree Search Algorithm
Tags: python,multithreading,binary

Question: I find implementing a multi-threaded binary tree search algorithm in Python can be challenging because it requires proper synchronization and management of multiple threads accessing shared data structures.
One approach, I think is to achieve this would be to use a thread-safe queue data structure to distribute search tasks to worker threads, and use locks or semaphores to ensure that each node in the tree is accessed by only one thread at a time.
How can you implement a multi-threaded binary tree search algorithm in Python that takes advantage of multiple cores, while maintaining thread safety and avoiding race conditions?

Answer: How can you implement a multi-threaded binary tree search algorithm in Python that takes advantage of multiple cores, while maintaining thread safety and avoiding race conditions?

You can write a multi-threaded binary tree search in Python that is thread-safe and has no race conditions.  Another answer makes some good suggestions about that.
But if you're writing it in pure Python then you cannot make effective use of multiple cores to improve the performance of your search, at least not with CPython, because the Global Interpreter Lock prevents any concurrent execution within the Python interpreter.  Multithreading can give you a performance improvement if your threads spend a significant fraction of their time in native code or blocked, but tree searching does not have any characteristics that would make room for an improvement from multithreading in a CPython environment.
Title: Error, index out of range. What is wrong?
Tags: python,python-3.x,algorithm

Question: I wrote a Python3 script to solve a picoCTF challenge. I received the encrypted flag which is:
cvpbPGS{c33xno00_1_f33_h_qrnqorrs}
From its pattern, I thought it is encoded using caesar cipher. So I wrote this script:
alpha_lower = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l',
        'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u','v', 'w', 'x', 'y', 'z']
alpha_upper = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L',
        'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z']
text = 'cvpbPGSc33xno00_1_f33_h_qrnqorrs '

for iterator in range(len(alpha_lower)):
    temp = ''
    for char in text:
        if char.islower():
        
            ind = alpha_lower.index(char)
            this = ind + iterator
            
            while this > len(alpha_lower):
                this -= len(alpha_lower)
                
            temp += alpha_lower[this]
            
        elif char.isupper():
            ind = alpha_upper.index(char)
            that = ind + iterator
            
            while that > len(alpha_upper):
                that -= len(alpha_upper)

            temp += alpha_upper[that]
    print(temp)


I understand what the error means. I can't understand where the flaw is to fix. Thanks in advance.
Sorrym here is the error:
Desktop>python this.py 
cvpbPGScxnofhqrnqorrs  
dwqcQHTdyopgirsorpsst
exrdRIUezpqhjstpsqttu
Traceback (most recent call last):
File "C:\Users\user\Desktop\this.py", line 18, in <module>
temp += alpha_lower[this]
IndexError: list index out of range

Answer: Why that break is simple :
If this==len(alpha_lower) then we won't enter your loop:
while this > len(alpha_lower):
And thus when trying temp += alpha_lower[this] it will return an error.
An index must be strictly inferior to the size of the array. Your condition should have been while this >= len(alpha_lower):.
As pointed out, a better method here is to use a modulus.
Title: pd.Grouper with datetime key in conjunction with another grouping key seemingly creates the wrong number of groups
Tags: python,pandas,datetime,group-by

Question: Using pd.Grouper with a datetime key in conjunction with another key creates a set of groups, but this does not seem to encompass all of the groups that need to be created, in my opinion.
>>> test = pd.DataFrame({"id":["a","b"]*3, "b":pd.date_range("2000-01-01","2000-01-03", freq="9H")})
>>> test
    id  b
0   a   2000-01-01 00:00:00
1   b   2000-01-01 09:00:00
2   a   2000-01-01 18:00:00
3   b   2000-01-02 03:00:00
4   a   2000-01-02 12:00:00
5   b   2000-01-02 21:00:00

When I tried to create groups based on the date and id values:
>>> g = test.groupby([pd.Grouper(key='b', freq="D"), 'id'])
>>> g.groups
{(2000-01-01 00:00:00, 'a'): [0], (2000-01-02 00:00:00, 'b'): [1]}

g.groups shows only 2 groups when I expected 4 groups: both "a" and "b" for each day.
However, when I created another column based off of "b":
>>> test['date'] = test.b.dt.date
>>> g = test.groupby(['date', 'id'])
>>> g.groups
{(2000-01-01, 'a'): [0, 2], (2000-01-01, 'b'): [1], (2000-01-02, 'a'): [4], (2000-01-02, 'b'): [3, 5]}

The outcome was exactly what I expected.
I don't know how to make sense of these different outcomes. Please enlighten me.

Answer: I believe it is because of the difference between 'pd.Grouper' and the 'dt.date' method in pandas. 'pd.Grouper' groups by a range of values (e.g., daily, hourly, etc.) while 'dt.date' returns just the date part of a datetime object, effectively creating a categorical variable.
When you use 'pd.Grouper' with a frequency of "D", it will group by full days, so each day is represented by only one group. But in your case, each id has multiple records for a given day. So, 'pd.Grouper' is not able to capture all of the groups that you expect.
On the other hand, when you use the 'dt.date' method to extract the date part of the datetime, it creates a categorical variable that represents each date independently.
so when you group by this new date column along with the id column, each group will correspond to a unique combination of date and id, giving you the expected outcome.
In summary, pd.Grouper is useful when you want to group by a range of values (e.g., daily, hourly), while using a separate column for the exact values (e.g., a column for dates only) is useful when you want to group by specific values.
Title: Exporting Certificates from AWS Certmanager Boto3 Python310
Tags: python-3.x,amazon-web-services,boto3

Question: So I manually imported a certificate and key pair issued by a third party to certmanage in AWS and I am trying to programaticly export to a webserver and I get this error:
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the ExportCertificate operation: Certificate ARN: arn:aws:acm:us-east-1:x:certificatexxxxxxxx is not a private certificate
Can I export a third party cert and private key from AWS certmanager?
python -V
Python 3.10.0
I am trying to export a AWS managed certificate from certmanager and its failing.
I've  tried googleing the error code but come up with nothing.

Answer: AWS Certificate Manager (ACM) has two types of certificates. Public and Private.
You can't export any certificate when it is public. Even if you imported it.
You can associate your ACM certificate with ALB, for example, and put this ALB in front of your EC2 instance. But you can't export.
As you imported the certificate, it means you have the public and private parts of the certificate. You can just use it on your instance.
Only ACM privates ones can be exported.
Title: RSI values in Python (lib is Pandas) don't match RSI values in Tradingview-Pinescript
Tags: python,pandas,tradingview-api,rsi

Question: I have a custom indicator that I use on Tradingview.  The values for the mst indicator in python do not match the values for mst indicator in Tradingview.  How do I fix this so the values are exactly the same?
The pinescript code is as follows:
//Calculate MST
RSI = ta.rsi(close, 14)
rsidelta    = ta.mom(RSI, 9)
rsisma      = ta.sma(ta.rsi(close, 3), 3)
mst         = rsidelta+rsisma
plot(mst, title="MST", color=#BB2BFA, linewidth = 2)

I am trying to replicate the exact values for MST in a python script.  The python code for RSI that I am using is as follows:
def rsi(df: pd.DataFrame, period: int = 14, source: str = 'close') -> np.array:
rsi = ta.rsi(df[source], period)
if rsi is not None:
    return rsi.values

This is code in my configuration file:
[scans.7] # MST
rsi_source = 'close'
rsi_period = 14
rsi_delta_period = 9
rsi_sma_period = 3
mst_threshold = [20, 80]

This is code in scanner.py file
              # Scan 7
                if '7' in self.scans:
                    scan = self.scans['7']
                    rsi = indicators.rsi(df=df, period=scan['rsi_period'], source=scan['rsi_source'])
                    rsi_delta = rsi[-1] - rsi[-scan['rsi_delta_period']]
                    rsi_sma = pd.Series(indicators.rsi(df=df, period=scan['rsi_sma_period'], source=scan['rsi_source'])).rolling(scan['rsi_sma_period']).mean()
                    mst = rsi_delta + rsi_sma

Answer: I have encountered the same issue on the EMA indicator for me the issue was that I took the last 500 candles from Binance and tried to calculate the EMA and see if the values matched Trading View's values, then I realized that the EMA + RSI indicators are recursive(meaning they relay on past result values to generate a result.) with this said it might be that the reason for this inaccuracy in results is simply the fact that my indicator calculation started at a different point than that of Trading view's resulting in slight inaccuracies between the results.
Title: Is it possible to run code without gui on android?
Tags: python,android

Question: I got python code that has no GUI and works in terminal. Can I convert it to apk and run on android?
I'm just curious if it's possible.

Answer: No, you cannot directly run a Python script in the terminal as an Android app. Python scripts are typically run on a computer using the Python interpreter, and Android devices use the Android operating system which is different from the typical computer operating systems.
However, you can use a tool such as Kivy, which is a Python library for creating mobile apps, to create an Android app from your Python script. Kivy provides a way to package your Python code into an Android app, so you can run it on an Android device.
I am sure there are other tools providing this option as well. These tools essentially bundle the Python interpreter and your script into a single executable file, so the user doesn't need to have Python installed on their device to run your app.
I believe there are tutorials on youtube as well so as to how to use Kivy to run your python code. I hope this helps :)
Title: Configuring Python execution from VS Code
Tags: python,python-3.x,visual-studio-code,logging,error-log

Question: I was trying learning about logging in python for the first time today. i discovered when i tried running my code from VS Code, i received this error message
/bin/sh: 1: python: not found however when i run the code directly from my terminal, i get the expected result. I need help to figure out the reason for the error message when i run the code directly from vscode
I've tried checking the internet for a suitable solution, no fix yet. i will appreciate your responses.

Answer: The error message you are receiving indicates that the "python" executable is not found in the PATH environment variable of the terminal you are using from within Visual Studio Code.
Add the location of the Python executable to the PATH environment variable in your terminal.
Specify the full path to the Python executable in your Visual Studio Code terminal.
You can find the full path to the Python executable by running the command "which python" in your terminal.
Title: does read method of io.BytesIO returns copy of underlying bytes data?
Tags: python,buffer,bytesio,memoryview

Question: I am aware that io.BytesIO() returns a binary stream object which uses in-memory buffer. but also provides getbuffer() which provides a readable and writable view (memoryview obj) over the contents of the buffer without copying them.
obj = io.BytesIO(b'abcdefgh')
buf = obj.getbuffer()

Now, we know buf points to underlying data and when sliced(buf[:3]) returns a memoryview object again without making a copy. So I want to know, if we do obj.read(3) does it also uses in-memory buffer or makes a copy ?. if it does uses in-memeory buffer, what is the difference between obj.read and buf and which one to prefer to effectively read the data in chunks for considerably very long byte objects ?

Answer: Simply put, BytesIO.read reads data from the in-memory buffer. The method reads the data and returns as bytes objects and gives you a copy of the read data. buf however, is a memory view object that views the underlying buffer and doesn't make a copy of the data.
The difference between BytesIO.read and buf is that, subsequent data retrieves will not be affected when io.BytesIO.read is used as you will get a copy of the data of the buffer, but if you change data bufyou also will change the data in the buffer as well.
In terms of performance, using obj.read would be a better choice if you want to read the data in chunks, because it provides a clear separation between the data and the buffer, and makes it easier to manage the buffer. On the other hand, if you want to modify the data in the buffer, using buf would be a better choice because it provides direct access to the underlying data.
Title: How to do the cross validation properly?
Tags: python,machine-learning,cross-validation,hyperparameters

Question: As an example, I can cross validation when I do hyperparameter tuning (GridSearchCV). I can select the best estimator from there and do RFECV. and I can perform cross validate again. But this is a time-consuming task. I'm new to data science and still learning things. Can an expert help me lean how to use these things properly in machine learning model building?
I have time series data. I'm trying to do hyperparameter tuning and cross validation in a prediction model. But it is taking a long time run. I need to learn the most efficient way to do these things during the model building process.

Answer: Cross-validation is a tool in order to evaluate model performance. Specifically avoid over-fitting. When we put all the data in training side, your Model will get over-fitting by ignoring generalisation of the data.
The concept of turning parameter should not based on cross-validation because hyper-parameter should be changed based on model performance, for example the depth of tree in a tree algorithm….
When you do a 10-fold cv, you will be similar to training 10 model, of cause it will have time cost. You could tune the hyper-parameter based on the cv result as cv-> model is a result of the model. However it does not make sense when putting the tuning and do cv to check again because the parameter already optimised based on the first model result.
P.s. if you are new to data science, you could learn something call regularization/dimension reduction to lower the dimension of your data in order to reduce time cost.
Title: Set a regex pattern to condition placing or removing spaces before or after a ) according to the characters that are before or after
Tags: python,regex

Question: import re

input_text = "((NOUN) )   ) de el auto rojizo, algo) ) )\n Luego ((PL_ADVB)dentro ((NOUN)de baúl ))abajo.) )."

input_text = input_text.replace(" )", ") ")

print(repr(input_text))

Simply using the .replace(" )", ") ") function I get this bad output, as it doesn't consider the conditional replacements that a function using regex patterns could, for example using re.sub( , ,input_text, flags = re.IGNORECASE)
'((NOUN))   )  de el auto rojizo, algo)) ) \n Luego ((PL_ADVB)dentro ((NOUN)de baúl) )abajo.)) .'

The goal is to get this output where closing parentheses are stripped of leading whitespace's and a single whitespace is added after as long as the closing parenthesis ) is not in front of a dot . , a newline \n or the end of line $
'((NOUN))) de el auto rojizo, algo)))\n Luego ((PL_ADVB)dentro ((NOUN)de baúl))abajo.)).'

Answer: Try this pattern it should solve it
/(\s*)())(\s*)(?=[^\s])/g
This pattern will match a ')' that is followed by a non-whitespace character and remove any spaces before or after the ')'.
If you want to add spaces around a ')' instead of removing them, you can modify the pattern like this:
/(\s*)())(\s*)(?=[^\s])/g
Title: Multiple replacements
Tags: python

Question: I want the user to be able to input more than one character they want to remove. It works but only if one character is entered.
string = input("Please enter a sentence: ")
removing_chars = input("Please enter the characters you would like to remove: ")
replacements = [(removing_chars, "")]

for char, replacement in replacements:
    if char in string:
        string = string.replace(char, replacement)

print(string)

Answer: when you loop over replacements, char takes removing_chars as value. Then, when you check if char in string, Python checks if removing_chars is a substring of string. To actually remove the characters separately, you have to loop over removing_chars in order to get the individual characters.
Title: FileNotFoundError with filepath that has whitespaces using ZipFile extract
Tags: python,path,python-zipfile

Question: I have a zip file with this structure:
Report 
│
└───folder1
│   │
│   └───subfolder1
|       |
│       │file 1 2022.txt
│   
└───folder2
    │   file2.txt

And their relative file paths are as follows: Report/folder1 / subfolder1 / file 1 2022.txt and Report/folder2/file2.txt
I tried to extract the zip file to another destination using the following code:
with ZipFile(attachment_filepath, 'r') as z:
    z.extractall('Destination')

However, it gives me a FileNotFoundError: [Winerror 3] The system cannot find the path specified: 'C:\\Users\\myname\\Desktop\\Report\\folder1 \\ subfolder1 '
I can extract just file2.txt without any problems but trying to extract file 1 2022.txt gives me that error,presumably due to all the extra whitespaces

Answer: "folder1 " (note the space) isn't the same as "folder1" (no space). When passing a path, it has to be the exact path. You can't add whitespace between path separators because the file system will assume you want a path name with spaces. Whatever put those spaces into the path is the problem.
Title: Issue importing application in Django in urls.html
Tags: python,django,django-views

Question: My src directory's layout is the following:

Learning

innit.py
settings.py
urls.py
wsgi.py


pages

innit.py
admin.py
apps.py
models.py
tests.py
views.py



Views.py has this code
from django.shortcuts import render
from django.http import HttpResponse

def home_view(*args,**kwargs):
    return HttpResponse("<h1>Hello World, (again)!</h1>")

urls.py has this code
from django.contrib import admin
from django.urls import path
from pages.views import home_view


urlpatterns = [
    path("", home_view, name = "home"),
    path('admin/', admin.site.urls),
]

The part where it says 'pages.views' in 'from pages.views import home_view' has a yellow/orange squiggle underneath it meaning that it is having problems importing the file and it just doesn't see the package/application called 'pages' and doesn't let me import it even though the package has a folder called 'innit.py'. Even worse is the fact that the tutorial I am currently following receives no such error and I can't see anyone else who has encountered this error.
As you probably expect I am a beginner so I don't have experience and this is my first time editing views.html in Django so I may have made an obvious mistake if so, just point it out.
I tried doing
from ..pages.views import home_view

However it failed and gave me an error
I have also tried changing the project root however this now causes issues with the imports in 'views.py'.

Answer: The part where it says 'pages.views' in 'from pages.views import home_view' has a yellow/orange squiggle underneath it meaning that it is having problems importing the file and it just doesn't see.
You need to mark the correct "source root". This is for Django the project directory, which is the directory that contains the apps.
For example in PyCharm you click right on that directory, and use Mark Directory as… ⟩ Sources Root.
Title: Is there a difference between math.inf and cmath.inf in Python?
Tags: python,python-3.x,complex-numbers,infinity,python-cmath

Question: Is there any difference between the infinities returned by the math module and cmath module?
Does the complex infinity have an imaginary component of 0?

Answer: Any difference?
No, there is no difference. According to the docs, both math.inf and cmath.inf are equivalent to float('inf'), or floating-point infinity.
If you want a truly complex infinity that has a real component of infinity and an imaginary component of 0, you have to build it yourself: complex(math.inf, 0)
There is, however, cmath.infj, if you want 0 as a real value and infinity as the imaginary component.
Constructing imaginary infinity
As others have pointed out math.inf + 0j is a bit faster than complex(math.inf, 0). We're talking on the order of nanoseconds though.
Title: Pandas dataframe sharing between functions isn't working
Tags: pandas,dataframe,python-asyncio

Question: I have a script that modifies a pandas dataframe with several concurrent functions (asyncio coroutines). Each function adds rows to the dataframe and it's important that the functions all share the same list. However, when I add a row with pd.concat a new copy of the dataframe is created. I can tell because each dataframe now has a different memory location as given by id().
As a result the functions are no longer share the same object. How can I keep all functions pointed at a common dataframe object?
Note that this issue doesn't arise when I use the append method, but that is being deprecated.

Answer: pandas dataframes are efficient because they use contiguous memory blocks, frequently of fundamental types like int and float. You can't just add a row because the dataframe doesn't own the next bit of memory it would have to expand into. Concatenation usually requires that new memory is allocated and data is copied. Once that happens, referrers to the original dataframe
If you know the final size you want, you can preallocate and fill. Otherwise, you are better off keeping a list of new dataframes and concatenating them all at once. Since these are parallel procedures, they aren't dependent on each others output, so this may be a feasable option.
Title: Python Polars consuming high memory and taking longer time
Tags: python,pandas,python-polars

Question: This is what I'm trying to do.

Scan the csv using Polars lazy dataframe
Format the phone number using a function
Remove nulls and duplicates
Write the csv in a new file

Here is my code
import sys
import json
import polars as pl
import phonenumbers

#define the variable and parse the encoded json
args = json.loads(sys.argv[1])

#format phone number as E164
def parse_phone_number(phone_number):
    try:
        return phonenumbers.format_number(phonenumbers.parse(phone_number, "US"), phonenumbers.PhoneNumberFormat.E164)
    except phonenumbers.NumberParseException:
        pass
    return None

#scan the csv file do some filter and modify the data and then write the output to a new csv file
pl.scan_csv(args['path'], sep=args['delimiter']).select(
    [args['column']]
).with_columns(
    #convert the int phne number as string and apply the parse_phone_number function
    [pl.col(args['column']).cast(pl.Utf8).apply(parse_phone_number).alias(args['column']),
    #add another column list_id with value 100
    pl.lit(args['list_id']).alias("list_id")
    ]
).filter(
    #filter nulls
    pl.col(args['column']).is_not_null()
).unique(keep="last").collect().write_csv(args['saved_path'], sep=",")

I tested a file with 800k rows and 23 columns (150mb) and it takes around 20 seconds and more than 500mb ram then it completes the task.
Is this normal? Can I optimize the performance (the memory usage at least)?
I'm really new with Polars and I work with PHP and I'm very noob at python too, so sorry if my code looks bit dumb haha.

Answer: You are using an apply, which means you are effectively writing a python for loop. This often is 10-100x slower than using expressions.
Try to avoid apply. And if you do use apply, don't expect it to be fast.
P.S. you can reduce memory usage by not casting the whole column to Utf8, but instead cast inside your apply function. Though I don't think using 500MB is that high. Ideally polars uses as much RAM as available without going OOM. Unused RAM might be wasted potential.
Title: PySpark- How to handle source data schema change
Tags: python,dataframe,apache-spark,pyspark,hive

Question: I'm trying to use PySpark to read from Avro file into dataframe, do some transformations and write the dataframe out to HDFS as hive tables using the code below. The file format for the hive tables is parquet.
df.write.mode("overwrite").format("hive").insertInto("mytable")
#this write a partition every day. When re-run, it would overwrite that run day's partition 

The problem is, when the source data has a schema change, like added a column, it will fail with an error saying: source file structure not match with existing table schema. How should I handle this case programmatically? Many thanks for your help.
Edited :I want the new schema changes to be reflected in target table. I'm looking for a programmatic way to do this.

Answer: You should be able to query off the system tables. You can run a comparison on these to see what changes have occurred since your last run.
Title: How do I make a Python request for WordPress REST API to create a custom field?
Tags: python,wordpress,rest,python-requests

Question: For example, I have a post and I want to update it with tags and some custom field, like 'rating' or 'mood' (not using any plugin, only WP built-in options for custom fields and REST API).
r = requests.post(WP_url, params = {'tags': tags, 'rating': rating}, auth = wp_auth)
Something like this. It works great for updating existing post parameters and fields, but I cannot find a way to create a custom field using Python API requests only.

Answer: I don't think it is possible to create a new field from request. Its depend on you WP Rest API Server, how it handles the excess argument you, if you API create a new field for any excess provided then only it will be possible to create new field.
Title: How do I store objects I created in np.array if a __getattr__ exists?
Tags: python,numpy-ndarray

Question: I created a Pixel class for image processing (and learn how to build a class). A full image is then a 2D numpy.array of Pixel but when I added a __getattr__ method , it stopped to work, because numpy wants an __array_struct__ attribute.
I tried to add this in __getattr__:
if name == '__array_struct__':
    return object.__array_struct__

Now it works but I get
'''DeprecationWarning: An exception was ignored while fetching the attribute __array__ from an object of type 'Pixel'.  With the exception of AttributeError NumPy will always raise this exception in the future.  Raise this deprecation warning to see the original exception. (Warning added NumPy 1.21)
I = np.array([Pixel()],dtype = Pixel)'''

a part of the class:
class Pixel:
    def __init__(self,*args):

        #things to dertermine RGB
        self.R,self.G,self.B = RGB
        
        #R,G,B are float between 0 and 255
    ...
    def __getattr__(self,name):
 
        if name == '__array_struct__':
            return object.__array_struct__
        if name[0] in 'iI':
            inted = True
            name = name[1:]
        else:
            inted = False
 
        if len(name)==1:
            n = name[0]

            if n in 'rgba':
                value = min(1,self.__getattribute__(n.upper())/255)
       
            elif n in 'RGBA':
                value = min(255,self.__getattribute__(n))
                assert 0<=value
            else:
                h,s,v = rgb_hsv(self.rgb)
                if n in 'h':
                    value = h
                elif n == 's':
                    value = s
                elif n == 'v':
                    value = v
                elif n == 'S':
                    value = s*100
                elif n == 'V':
                    value = v*100
                elif n == 'H':
                    value = int(h)
            if inted:
                return int(value)
            else:
                return value
        else:
            value = []
            for n in name:
                try:
                    v = self.__getattribute__(n)
                except AttributeError:
                    v = self.__getattr__(n)
                if inted:
                    value.append(int(v))
                else:
                    value.append(v)
            return value

Answer: Your class should either implement __array__ or raise an AttributeError when numpy tries to get it. The warning message says you raised some other error and that numpy will not accept that in the future. I haven't figured out your code well enough to know, but it could be that calling self.__getattr__(n) inside of __getattr__ hits a maximum recursion error.
object.__array_struct__ doesn't exist and so just by luck its AttributeError exception is what numpy was looking for. A better strategy is to raise AttributeError for anything that doesn't meet the selection criteria for your automatically generated attributes. Then you can take out the special case for __array_struct__ that doesn't work properly anyway.
Title: Functions depending on other functions in Python
Tags: python

Question: def mean(x):
    return(sum(x)/len(x))

def variance(x):
    x_mean = mean(x)
    return sum((x-x_mean)**2)/(len(x)-1)

def standard_deviation(x):
    return math.sqrt(variance(x))

The functions above build on each other. They depend on the previous function. What is a good way to implement this in Python? Should I use a class which has these functions? Are there other options?

Answer: Because they are widely applicable, keep them as they are
Many parts of a program may need to calculate these statistics, and it will save wordiness to not have to get them out of a class. Moreover, the functions actually don't need any class-stored data: they would simply be static methods of a class. (Which in the old days, we would have simply called "functions"!)
If they needed to store internal information to work correctly, that is a good reason to put them into a class
The advantage in that case is that it is more obvious to the programmer what information is being shared. Moreover, you might want to create two or more instances that had different sets of shared data. That is not the case here.
Title: How can I get quantity with SP API Python
Tags: python,amazon-selling-partner-api

Question: I need a product's unit of stock(quantity). Is it possible with SP API, if possible how can I get it?
Note: I can get it with SKU like the following code but the product is not listed by my sellers.
from sp_api.api import Inventories
quantity = Inventories(credentials=credentials, marketplace=Marketplaces.FR).get_inventory_summary_marketplace(**{
        "details": False,
        "marketplaceIds": ["A13V1IB3VIYZZH"],
        "sellerSkus": ["MY_SKU_1" , "MY_SKU_2"]
})
print(quantity)

Answer: from sp_api.api import Inventories
quantity = Inventories(credentials=credentials, marketplace=Marketplaces.FR).get_inventory_summary_marketplace(**{
"details": False,
"marketplaceIds": ["A13V1IB3VIYZZH"],
"sellerSkus": ["MY_SKU_1" , "MY_SKU_2"]
})
print(quantity)
Title: AttributeError: 'Adam' object has no attribute 'get_updates'
Tags: python,tensorflow

Question: I'm training a VAE with TensorFlow Keras backend and I'm using Adam as the optimizer. the code I used is attached below.
    def compile(self, learning_rate=0.0001):
    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
    self.model.compile(optimizer=optimizer,
                       loss=self._calculate_combined_loss,
                       metrics=[_calculate_reconstruction_loss,
                                calculate_kl_loss(self)])

The TensorFlow version I'm using is 2.11.0. The error I'm getting is
AttributeError: 'Adam' object has no attribute 'get_updates'

I'm suspecting the issues arise because of the version mismatch. Can someone please help me to sort out the issue? Thanks in advance.

Answer: Of late, I had to use the tensorflow2.5 and I replaced all "import keras" by "import tensorflow.keras".
Now I use tensorflow2.12 and I met this error and when I returned those replacements; this error was removed.
thank you!
Title: AttributeError: 'Adam' object has no attribute 'get_updates'
Tags: python,tensorflow

Question: I'm training a VAE with TensorFlow Keras backend and I'm using Adam as the optimizer. the code I used is attached below.
    def compile(self, learning_rate=0.0001):
    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
    self.model.compile(optimizer=optimizer,
                       loss=self._calculate_combined_loss,
                       metrics=[_calculate_reconstruction_loss,
                                calculate_kl_loss(self)])

The TensorFlow version I'm using is 2.11.0. The error I'm getting is
AttributeError: 'Adam' object has no attribute 'get_updates'

I'm suspecting the issues arise because of the version mismatch. Can someone please help me to sort out the issue? Thanks in advance.

Answer: Two ways worked for me,

By using  tf.keras.optimizers.legacy.SGD - instead of  tf.keras.optimizers.SGD

Importing statement is changed from
import tensorflow.keras as keras to 'import keras'
Title: Regex - removing everything after first word following a comma
Tags: python,regex

Question: I have a column that has name variations that I'd like to clean up. I'm having trouble with the regex expression to remove everything after the first word following a comma.
d = {'names':['smith,john s','smith, john', 'brown, bob s', 'brown, bob']}
x = pd.DataFrame(d)

Tried:
x['names'] =  [re.sub(r'/.\s+[^\s,]+/','', str(x)) for x in x['names']]

Desired Output:
['smith,john','smith, john', 'brown, bob', 'brown, bob']

Not sure why my regex isn't working, but any help would be appreciated.

Answer: Try re.sub(r'/(,\s*\w+).*$/','$1', str(x))...
Put the triggered pattern into capture group 1 and then restore it in what gets replaced.
Title: make 3d numpy array using for loop in python
Tags: python,arrays,numpy,3d,2d

Question: I have training data with 2 dimension. (200 results of 4 features)
I proved 100 different applications with 10 repetition resulting 1000 csv files.
I want to stack each csv results for machine learning.
But I don't know how.
each of my csv files look like below.
test1.csv to numpy array data
[[0 'crc32_pclmul' 445 0]
 [0 'crc32_pclmul' 270 4096]
 [0 'crc32_pclmul' 234 8192]
 ...
 [249 'intel_pmt' 272 4096]
 [249 'intel_pmt' 224 8192]
 [249 'intel_pmt' 268 12288]]

I tried below python code.
path = os.getcwd()
csv_files = glob.glob(os.path.join(path, "*.csv"))
cnt=0
for f in csv_files:
    cnt +=1
    seperator = '_'
    app = os.path.basename(f).split(seperator, 1)[0]

    if cnt==1:
        a = np.array(preprocess(f))
        b = np.array(app)
    else:
        a = np.vstack((a, np.array(preprocess(f))))
        b = np.append(b,app)
print(a)
print(b)

preprocess function returns df.to_numpy results for each csv files.
My expectation was like below. a(1000, 200, 4)
[[[0 'crc32_pclmul' 445 0]
 [0 'crc32_pclmul' 270 4096]
 [0 'crc32_pclmul' 234 8192]
 ...
 [249 'intel_pmt' 272 4096]
 [249 'intel_pmt' 224 8192]
 [249 'intel_pmt' 268 12288]],
[[0 'crc32_pclmul' 445 0]
 [0 'crc32_pclmul' 270 4096]
 [0 'crc32_pclmul' 234 8192]
 ...
 [249 'intel_pmt' 272 4096]
 [249 'intel_pmt' 224 8192]
 [249 'intel_pmt' 268 12288]],
...
[[0 'crc32_pclmul' 445 0]
 [0 'crc32_pclmul' 270 4096]
 [0 'crc32_pclmul' 234 8192]
 ...
 [249 'intel_pmt' 272 4096]
 [249 'intel_pmt' 224 8192]
 [249 'intel_pmt' 268 12288]]]

However, I'm getting this. a(200000, 4)
[[0 'crc32_pclmul' 445 0]
 [0 'crc32_pclmul' 270 4096]
 [0 'crc32_pclmul' 234 8192]
 ...
 [249 'intel_pmt' 272 4096]
 [249 'intel_pmt' 224 8192]
 [249 'intel_pmt' 268 12288]]

I want to access each csv results using a[0] to a[1000] each sub-array looks like (200,4)
How can I solve the problem? I'm quite lost

Answer: Make a new list (outside of the loop) and append each item to that new list after reading.
Title: How to install Python or R in an apptainer?
Tags: python,docker,apptainer

Question: I am new to docker and using apptainer for that.
the def file is: firstApp.def:
`Bootstrap: docker
From: ubuntu:22.04

%environment
    export LC_ALL=C
`

then I built it as follows and I want it to be writable (I hope I am not so naive), so I can install some packages later:
`apptainer build --sandbox --fakeroot firstApp.sif firstApp.def
`

now I do not know how to install Python3 (preferably, 3.8 or later).
I tried to add the following command lines to the def file:
`%post
   apt-get -y install update
   apt-get -y install python3.8 `

it raises these errors as well even without "apt-get -y install python3.8":
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package update
FATAL:   While performing build: while running engine: exit status 100

Answer: This work for me
%post
apt-get update && apt-get install -y netcat python3.8
Title: Python code yielding different result for same numerical value, depending on inclusion of precision point
Tags: python-3.x,numpy,scipy

Question: I defined a function which returns a third order polynomial function for either a value, a list or a np.array:
def two_d_third_order(x, a, b, c, d):
    return a + np.multiply(b, x) + np.multiply(c, np.multiply(x, x)) + np.multiply(d, np.multiply(x, np.multiply(x, x)))

The issue I noticed is, however, when I use "two_d_third_order" on the following two inputs:
1500
1500.0
With (a, b, c, d) = (1.20740028e+00, -2.93682465e-03,  2.29938078e-06, -5.09134552e-10), I get two different results:
2.4441
0.2574
, respectively. I don't know how this is possible, and any help would be appreciated.
I tried several inputs, and somehow the inclusion of a floating point on certain values (despite representing the same numerical value) changes the end result.

Answer: Python uses implicit data type conversions. When you use only integers (like 1500), there is a loss of precision in all subsequent operations. Whereas when you pass it a float or double (like 1500.0), subsequent operations are performed with the associated datatype, i.e in this case with higher precision.
This is not a "bug" so to speak, but generally how Python operates without the explicit declaration of data types. Languages like C and C++ require explicit data type declarations and explicit data type casting to ensure operations are performed in the prescribed precision formats. Can be a boon or a bane depending on usage.
Title: Trouble shooting using Pythonnet and setting Runtime.PythonDLL property
Tags: python,.net,clr,python.net

Question: I try to use an assembly for .NET framework 4.8 via Pythonnet. I am using version 3.0.1 with Python 3.10. The documentation of Pythonnet is stating:

You must set Runtime.PythonDLL property or PYTHONNET_PYDLL environment variable starting with version 3.0, otherwise you will receive BadPythonDllException (internal, derived from MissingMethodException) upon calling Initialize. Typical values are python38.dll (Windows), libpython3.8.dylib (Mac), libpython3.8.so (most other Unix-like operating systems).

However, the documentation unfortunately is not stating how the property is set and I do not understand how to do this.
When I try:
import clr
from pythonnet import load

load('netfx')

clr.AddReference(r'path\to\my.dll')

unsurprisingly the following error is coming up
Failed to initialize pythonnet: System.InvalidOperationException: This property must be set before runtime is initialized
   bei Python.Runtime.Runtime.set_PythonDLL(String value)
   bei Python.Runtime.Loader.Initialize(IntPtr data, Int32 size)
   bei Python.Runtime.Runtime.set_PythonDLL(String value)
   bei Python.Runtime.Loader.Initialize(IntPtr data, Int32 size)
[...]
in load
    raise RuntimeError("Failed to initialize Python.Runtime.dll")
RuntimeError: Failed to initialize Python.Runtime.dll

The question now is, where and how the Runtime.PythonDLL property or PYTHONNET_PYDLL environment variable is set
Thanks,
Jens

Answer: I believe this is because import clr internally calls pythonnet.load, and in the version of pythonnet you are using this situation does not print any warning.
E.g. the right way is to call load before you call import clr for the first time.
Title: Python entry_point in virtual environment not working
Tags: python,package,virtualenv,setuptools,entry-point

Question: I have a virtual environment where I am developing a Python package. The folder tree is the following:
working-folder
|-setup.py
|-src
  |-my_package
    |-__init__.py
    |-my_subpackage
      |-__init__.py
      |-main.py

main.py contains a function my_main that ideally, I would want to run as a bash command.
I am using setuptools and the setup function contains the following line of code
setup(
...
    entry_point={
        "console_scripts": [
            "my-command = src.my_package.my_subpackage.main:my_main",
        ]
    },
...
)


When I run pip install . the package gets correctly installed in the virtual environment. However, when running my-command on the shell, the command does not exist.
Am I missing some configuration to correctly generate the entry point?

Answer: I simply mistyped the argument entry_point, which actually is entry_points. Unfortunately, I was not getting any output errors.
Title: How to disable the Matplotlib navigation toolbar in a particular axis?
Tags: python,matplotlib,user-interface,widget,interactive

Question: I have a figure with different plots on several axes. Some of those axes do not play well with some of the navigation toolbar actions. In particular, the shortcuts to go back to the home view and the ones to go to the previous and next views.
Is there a way to disable those shortcuts only for those axes? For example, in one of the two in the figure from the example below.
import matplotlib.pyplot as plt

# Example data for two plots
x1 = [1, 2, 3, 4]
y1 = [10, 20, 25, 30]
x2 = [2, 3, 4, 5]
y2 = [5, 15, 20, 25]

# Create figure and axes objects
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))

# Plot data on the first axis
ax1.plot(x1, y1)
ax1.set_title("First Plot")

# Plot data on the second axis
ax2.plot(x2, y2)
ax2.set_title("Second Plot")

# Show plot
plt.show()


Edit 1:
The following method will successfully disable the pan and zoom tools from the GUI toolbox in the target axis.
ax2.set_navigate(False)

However, the home, forward, and back buttons remain active. Is there a trick to disable also those buttons in the target axis?

Answer: You can try to use ax2.get_xaxis().set_visible(False)
Title: How to find and click the "Like" button on Facebook page using Selenium
Tags: python,selenium,selenium-webdriver,xpath,nosuchelementexception

Question: I am trying to automate the process of liking pages on Facebook. I've got a list of each page's link and I want to open and like them one by one.
I think the Like button doesn't have any id or name, but it is in a span class.
<span class="x1lliihq x6ikm8r x10wlt62 x1n2onr6 xlyipyv xuxw1ft">Like</span>

I used this code to find and click on the "Like" button.
def likePages(links, driver):
    for link in links:
        driver.get(link)
        time.sleep(3)
        driver.find_element(By.LINK_TEXT, 'Like').click()

And I get the following error when I run the function:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element

Answer: You cannot use Link_Text locator as Like is not a hyperlink. Use XPath instead, see below:
XPath : //span[contains(text(),"Like")]
driver.find_element(By.XPATH, '//span[contains(text(),"Like")]').click()
Title: how to automatically generate mypy stubs using pyproject.toml
Tags: python,python-3.x,mypy,pyproject.toml

Question: i have a package and in it i use pyproject.toml
and for proper typing i need stubs generated, although
its kinda annoying to generate them manually every time,
so, is there a way to do it automatically using it ?
i just want it to run stubgen and thats it, just so
mypy sees the stubs and its annoying seeing linters
throw warnings and you keep having to # type: ignore
heres what i have as of now, i rarely do this so its probably
not that good :
[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[project]
name = "<...>"
authors = [
    {name = "<...>", email = "<...>"},
]
description = "<...>"
readme = "README"
requires-python = ">=3.10"
keywords = ["<...>"]
license = {text = "GNU General Public License v3 or later (GPLv3+)"}
classifiers = [
    "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
    "Programming Language :: Python :: 3",
]
dependencies = [
    "<...>",
]
dynamic = ["version"]


[tool.setuptools]
include-package-data = true

[tool.setuptools.package-data]
<...> = ["*.pyi"]

[tool.pyright]
pythonVersion = "3.10"
exclude = [
    "venv",
    "**/node_modules",
    "**/__pycache__",
    ".git"
]
include = ["src", "scripts"]
venv = "venv"
stubPath = "src/stubs"
typeCheckingMode = "strict"
useLibraryCodeForTypes = true
reportMissingTypeStubs = true

[tool.mypy]
exclude = [
    "^venv/.*",
    "^node_modules/.*",
    "^__pycache__/.*",
]

thanks for the answers in advance

Answer: just make a shellscript and add it to pyproject.toml as a script
:+1:
Title: .exe file opening Powershell Window
Tags: python,powershell,pyinstaller,exe

Question: I made an .exe file using pyinstaller, but when I run the file it opens a PowerShell window as well. I was wondering if there is anyway I can get it to not open so I just have the python program open.
I haven't really tried anything as I don't really know what I'm doing.

Answer: if you run it from terminal, you can use this command:
start /min "" "path\file_name.exe"
Title: .exe file opening Powershell Window
Tags: python,powershell,pyinstaller,exe

Question: I made an .exe file using pyinstaller, but when I run the file it opens a PowerShell window as well. I was wondering if there is anyway I can get it to not open so I just have the python program open.
I haven't really tried anything as I don't really know what I'm doing.

Answer: When running pyinstaller be sure to use the --windowed argument. For example:

pyinstaller –-onefile myFile.py –-windowed