Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Airflow SSHOperator Command Timed Out when executing Python script
Tags: python,ssh,airflow

Question: I created a DAG that successfully uses SSHOperator to execute a simple Python script from a server(note I have set cmd_timeout = None
When I change the simple Python script to a more complex script, I get an error for  "SSH command timed out"
Additionally, if i log into the server, open CMD and execute C:/ProgramData/Python/Scripts/activate.bat && python C:/Users/Main/Desktop/Python/Python_Script.py, which is the one that "times out". It is successful, so I don't believe it is an issue with script or access.
Is there an additional setting that must changed to avoid the SSH command time out when executing commands?
Log:
[2023-03-09, 21:31:53 UTC] {ssh.py:123} INFO - Creating ssh_client
[2023-03-09, 21:31:53 UTC] {ssh.py:101} INFO - ssh_hook is not provided or invalid. Trying ssh_conn_id to create SSHHook.
[2023-03-09, 21:31:53 UTC] {base.py:73} INFO - Using connection ID 'server' for task execution.
   [2023-03-09, 21:31:53 UTC] {transport.py:1874} INFO - Connected (version 2.0, client OpenSSH_for_Windows_8.1)
[2023-03-09, 21:31:53 UTC] {transport.py:1874} INFO - Authentication (password) successful!
[2023-03-09, 21:31:53 UTC] {ssh.py:465} INFO - Running command: call C:/ProgramData/Python/Scripts/activate.bat && python C:/Users/Main/Desktop/Python/Python_Script.py
[2023-03-09, 21:32:03 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 158, in execute
    result = self.run_ssh_client_command(ssh_client, self.command)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 143, in run_ssh_client_command
    exit_status, agg_stdout, agg_stderr = self.ssh_hook.exec_ssh_client_command(
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/ssh/hooks/ssh.py", line 526, in exec_ssh_client_command
    raise AirflowException("SSH command timed out")
airflow.exceptions.AirflowException: SSH command timed out

Answer: I was able to solve this.  I had to change cmd_timeout = None to cmd_timeout = 7200.  You could choose a different arbitrary large number.
Title: Anaconda 3 not installing pyodbc v5
Tags: python,anaconda,pyodbc,miniconda

Question: I have installed Anaconda 3 which includes Python 3.9.16. When I install pyodbc in miniconda it is only version 4. I read it needs to be version 5 if Python is above 3.7. When I run my Python script it I get the message
ModuleNotFoundError: No module named 'pyodbc' even though it is installed.
Is this because pyodbc is version 4 and should be version 5 or could it be another reason? I've tried all of the different flavours of installing pyodbc via Conda but I still get this error. I do also have Python 3.11 installed manually. Could there be a conflict?

Answer: In the end, I installed Python 3.9 separately to the Anaconda install and I had to install pyodbc using the pip command and it worked. I assumed Anaconda used a different install of Python and would require a conda install.
Title: Pydash: how to find using a object: py_.collections.find(DATA, OBJECT)
Tags: python,lodash,pydash

Question: In lodash I can use the syntax:
find(ARRAY_OF_OBJECTS, OBJECT)
This will return an object from the array if it meets the criteria of the passed object. In this case OBJECT would be e.g. { active: true, dimension: 'target' }. The objects in the array would contain e.g. active, dimension, status etc.
How can I do the same in pydash? I know I can do find(ARRAY_OF_OBJECTS, lambda x: x.active == True, but the thing is, the object I pass is dynamically made. So sometimes it might not have active (as example)

Answer: Figured it out. I can do it with is_match from pydash. In a complete line of code it would become this. target_data is an array of objects and source_row['dimensions'] is an object
py_.collections.find(target_data, lambda x: py_.predicates.is_match(x, source_row['dimensions']))
Title: How do I set simple linear constraints with dual_annealing?
Tags: python,scipy,mathematical-optimization

Question: I can set simple bounds to use with dual_annealing: E.g.
upper_bound = 20
num_points = 30
bounds = [(0, upper_bound) for i in range(num_points)]
res = dual_annealing(fun, bounds, maxiter=1000)

But I would also like to constrain the variables so that x_i >= x_{i-1}+0.5 for each i. That is each variable should be at least 0.5 larger than the one preceding it.
How can you do that?
If scipy can't do it, are there other libraries with global optimizers that can?

Answer: You could try using a Lagrangian multiplier to penalize the objective function if your constraints aren't satisfied. Alternatively you can use the differential_evolution minimiser.
Title: Joining 2 dataframes and collecting only the unique values in pyspark
Tags: python,dataframe,pyspark

Question: I have 2 dataframes, the first one that is called questions_version1 and the second one called questions_version2.
questions_version1
+-----------+----------+-----------+--------------+
|  question |answer    |version    | total_answers
+-----------+----------+-----------+--------------+
|eye color  | blue     |  1        |  15000       |
|eye color  | brown    |  1        |  32000       |
|eye color  | green    |  1        |  5000        |
|hair color | brown    |  1        |  47000       |
|hair color | blonde   |  1        |  3000        |
|hair color | white    |  1        |  2000        |
+-----------+----------+-----------+--------------+


questions_version2
+-----------+----------+-----------+--------------+
|  question |answer    |version    | total_answers
+-----------+----------+-----------+--------------+
|eye color  | blue     |  1        |  15000       |
|eye color  | green    |  1        |  5000        |
|eye color  | hazel    |  2        |  9000        |
|hair color | brown    |  1        |  47000       |
|hair color | white    |  1        |  2000        |
|hair color | red      |  2        |  500         |
+-----------+----------+-----------+--------------+


How do I join both so I can get all the values in questions_version2 that doesn't exist in questions_version1 without duplicating all the repeated values that exist in both?
The final result would be something like this:
questions_1_and_2_merged
+-----------+----------+-----------+--------------+
|  question |answer    |version    | total_answers
+-----------+----------+-----------+--------------+
|eye color  | blue     |  1        |  15000       |
|eye color  | brown    |  1        |  32000       |
|eye color  | green    |  1        |  5000        |
|eye color  | hazel    |  2        |  9000        |
|hair color | brown    |  1        |  47000       |
|hair color | blonde   |  1        |  3000        |
|hair color | white    |  1        |  2000        |
|hair color | red      |  2        |  500         |
+-----------+----------+-----------+--------------+


Could you all help me with this please? Thanks!

Answer: I think you could loop through the second list and do:

if elem in questions_version1:
continue
else:
questions_version1.append(elem)
Title: I am not getting an option to close the window in pygame
Tags: python,pygame

Question: So, this is a simple code written to make the window and the background image. I need help with 2 things. 1. I am not getting the option to close, minimize or maximize the window, the entire thing is covered with my background image. 2. I need help in not making my image so bloated, or stretched. It is quite high-res but I believe I have made it stretched.
Code is below:
import pygame as p
import time
import random
import sys

# Defining global variables

HEIGHT = 800
WIDTH = 1000

# Adding a background image
BACKGROUND = p.image.load("background_image.jpg")


def draw():
    WINDOW.blit(BACKGROUND, (0, 0))
    p.display.update()


# Making the window
WINDOW = p.display.set_mode((WIDTH, HEIGHT))
p.display.set_caption("Space Dodge")


# Window will quit after pressing the X
def main():
    run = True
    while run:
        for event in p.event.get():
            if event.type == p.QUIT:
                run = False
                break

        WINDOW.fill((0, 0, 0))
        draw()

    p.quit()
    sys.exit()


if __name__ == "__main__":
    main()

Answer: Ther is a problem with your height of the screen.
The Option are available but it is not visible in you screen.
Solution : Reduce the HEIGHT and run the code Sure It will work.
Title: How to keep `LD_LIBRARY_PATH` from getting overwritten to pwd when running python script
Tags: linux,bash,python,environment-variables,shared-libraries

Question: I have a python script which eventually calls a binary that requires certain shared libraries. However, I have been running into the following error:
error while loading shared libraries: libmkl_rt.so: cannot open shared object file: No such file or directory

I have narrowed down the cause of this error to the fact that python overwrites LD_LIBRARY_PATH to be the PWD. When executing the binary manually via the bash, everything works properly as this environment variable is properly set. Here are examples of the current behavior:
$ echo $LD_LIBRARY_PATH
/the/proper/paths

$ python
>>> import os
>>> os.environ.get('LD_LIBRARY_PATH')
/the/proper/paths

$ python script.py
/my/present/working/directory

Here, script.py has the same contents as the second example.
Other information asked for:
$ which python
~/.pyenv/shims/python
$ python --version
Python 3.7.3
$ type python
python is hashed (/home/ec2-user/.pyenv/shims/python)
$ realpath `which python`
/home/ec2-user/.pyenv/shims/python

I would like to know if there's a way to see where this variable is getting changed and/or why this would be happening.
I am aware that I could simply hard-code the in the script, but that is a band-aid solution, I would rather know exactly what is happening under the hood.

Answer: I am aware that I could simply hard-code the in the script, but that is a band-aid solution, I would rather know exactly what is happening under the hood.

What's happening under the hood is that some code in your script.py (or code which script.py imports) sets the environment variable.
You can run gdb --args python script.py, set a breakpoint on setenv and putenv, and have GDB stop the program when the environment is reset. However, this will stop somewhere in the Python runtime, and figuring out which Python code is running might get tricky.

That said, setting LD_LIBRARY_PATH is not the right way to fix the "binary that requires certain shared libraries" problem in the first place.
A much better solution is to either link the binary with correct -rpath flag, or to fix already-linked binary using patchelf --set-rpath, so the binary runs correctly without LD_LIBRARY_PATH.
Title: Why some Django view classes had not been defined as abstract base class?
Tags: python,python-3.x,django,django-views,abstract-class

Question: I'm writing a small and lightweight Django-like back-end framework for myself just for experiment.
If we look at ProcessFormView view(and some other views):
class ProcessFormView(View):
    def get(self, request, *args, **kwargs):
        return self.render_to_response(self.get_context_data())

    def post(self, request, *args, **kwargs):
        form = self.get_form()
        if form.is_valid():
            return self.form_valid(form)
        else:
            return self.form_invalid(form)

    ...

To me it sounds like a valid case to define this class as an "abstract base class".
After all, It needs that sub-classes provide render_to_response(), get_context_data(), get_form(), form_valid(), form_invalid(). (They will be provided by TemplateResponseMixin and FormMixin.)
I can do something like this:
class ProcessFormView(View, metaclass=ABCMeta):
    @abstractmethod
    def render_to_response(self, context):
        pass

    @abstractmethod
    def get_context_data(self):
        pass

    @abstractmethod
    def get_form(self):
        pass

    @abstractmethod
    def form_valid(self, form):
        pass

    @abstractmethod
    def form_invalid(self, form):
        pass

    def get(self, request, *args, **kwargs):
        return self.render_to_response(self.get_context_data())

    def post(self, request, *args, **kwargs):
        form = self.get_form()
        if form.is_valid():
            return self.form_valid(form)
        else:
            return self.form_invalid(form)

    ...

Or even better we can factor these abstract methods out to another ABC class and inherit from it to clear things up.
I know that of course it's a made decision and nothing is wrong with it. I mostly interested to know if do what I just showed, how can it cause problem in future? what would be the cons that I'm not aware of?
The only disadvantage that I can think about is I should then write many abstract classes! This make the code base much bigger.

Answer: Aside from the mentioned fact that the code base would become much bigger, I think I found the main reason.
It's true that if we subclass ProcessFormView and "intend to use its get and post method", we have to eventually somehow provide those mentioned methods. But what if for some reason we only want to use its post method and provide our own custom method for get? (They are only get called inside post and get).
This way we no longer have to implement the render_to_response and get_context_data() abstract method. Defining them as abstractmethods would unnecessarily force sub-classes to implement them.
Title: How to save fingerprint data in django
Tags: python,django,django-models,django-views,fingerprint

Question: So I have a project where I have a model called Beneficiary where I need to store the fingerprint.
This is the model:
class Beneficiary(models.Model):
    image=models.ImageField(upload_to='home/images/',blank=True,null=True)
    name = models.CharField(max_length=200)
    gender=models.CharField(max_length=6,choices=GENDER_CHOICES,default='male')
    dob=models.DateField()
    registration_date=models.DateField()

I want to save the fingerprints of the beneficiaries in the model.
I am using DigitalPersona2 fingerprint sensor.
I am pretty new to fingerprints so can anyone help?

Answer: I don't know much about DigitalPersona2, but generally, the Fingerprint device should come with an SDK, that must be installed on the system that uses the fingerprint scanner device. DigitalPersona2 would have an SDK API that would allow you to communicate with the device.
Title: Guppy install in Python 3.11 fails because of missing longintrepr.h file
Tags: python,pip,guppy

Question: Operating System: Windows
I try:
pip install guppy3

It runs successfully at first but fatally fails towards the end.
The error message is:
Collecting guppy3
  Using cached guppy3-3.1.2.tar.gz (335 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: guppy3
  Building wheel for guppy3 (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [95 lines of output]

     //so long so i delete it

      bitset.c
      src/sets/bitset.c(7): fatal error C1083: 無法開啟包含檔案: 'longintrepr.h': No such file or directory
      error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.34.31933\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2

By the way I have all ready update
wheel 0.38.4
pip   23.0.1
setuptools  67.6.0
I really have no clue...
Hope someone who can help me to fix this error!

Answer: I find the solution!I just downgrade my python version to 3.9
Title: How to dynamically assign variable into #define statement in header file
Tags: c,header-files,micropython

Question: I have EXTREMELY limited programming experience but am working on a custom Micropython build. A feature is enabled/disabled using a #define statement in a C header file e.g. #define MICROPY_HW_USB_MSC (1); I would like to make this dynamic as part of the code build.
This defines whether the MCU supports USB Mass Storage at runtime. What I would like to do is to set this variable (?) from a script (or some other recommended means) that reads the state of a GPI pin on the MCU at boot.
So if the pin is low, it effectively creates #define MICROPY_HW_USB_MSC (0), and conversely if the pin is high, #define MICROPY_HW_USB_MSC (1).
I hope this makes sense and thanks in advance for any assistance.
I have tried searching for a solution online but am not experienced enough to search for the correct terminology - hence turning to experts for assistance.

Answer: Anything specified in #define's is done by the preprocessor that runs at compile time. This means it cannot know what the state of a pin is at runtime.
The only way to make a feature dynamic is to write actual C code that checks the state of the pin and acts on it.
Title: Error on android using uiautomator2 on Pydroid3
Tags: python,android,android-uiautomator,pydroid

Question: I'm trying to automate tasks on Android, but when I run my code on Pydroid3 on Android, it return me a error...
My device is a Asus Zenfone Selfie 4
I'dont understand this error because the error is about a ADB EXE problem.
Android have ADB EXE?
The code:
#MODULES IMPORT
import os
import cv2
import numpy as np
from uiautomator2 import Device
from uiautomator2 import connect

#CLASS AND FUNCTIONS DEFINE
class AndroidAuto:
    def _init_ (self, Serial = "Serial"):
        self.Serial = Serial

    def Android_Screenshot(self, dir = ".screenshots"):
        dir = dir

        if not os.path.exists(dir):
            os.makedirs(dir)

        device = Device(self.Serial)
        device.screenshot(f"{dir}/.screenshot_opencv.png")
        return (f"{dir}/.screenshot_opencv.png")

    def Android_Touch(self, x = 0, y = 0):
        device = Device(self.Serial)
        device.click(x, y)

    def Android_Get_Position(self, target = None):
        Device.toast("Starting ")
        while True:
            template = AndroidAuto().Android_Screenshot()

            w, h = target.shape[:-1]

            res = cv2.matchTemplate(template, target, cv2.TM_CCOEFF_NORMED)
            threshold = 0.8
            loc = np.where(res >= threshold)

            if len(loc[0]) > 0:
                x, y = loc[::-1]
                return x, y
            else:
                x, y = None, None
                Device.toast("Trying again!")
                continue

    def Android_Find_Click(self, target = "", x = 0, y = 0):
        while True:
            match = AndroidAuto().Android_Get_Position(target)

            if match != None:
                AndroidAuto().Android_Touch(x, y = match)
                break
            else:
                continue

AA = AndroidAuto()

print(AA.Android_Screenshot())

Then when I play, it returns me this is the error:
RuntimeError: No adb exe could be found. install adb on your system

Answer: That's what site description says:

python-uiautomator2 is a python-wrapper, which allows:

scripting with Python on computer
controlling the mobile with computer without usb connection.


According to this (and my guess) uiautomator2 relies on ADB.exe to connect to the device, searches for it in $PATH system variable, fails to find it and shows you an error.
So I guess it's not possible to use uiautomator2 in Pydroid3 like you would on PC.
Title: Anagram function returns False for unknown reason in some words - Python
Tags: python,anagram

Question: I've tried to create a function that returns True if two words are anagrams (have the same letters) to each other.
I realized that my counter counts to 13 but not sure what is the issue.
def is_anagram (worda: str , wordb: str) -> bool:
    count_same_letters = 0
    if len(worda) == len(wordb):
        for i in worda:
            for j in wordb:
                if i == j:
                    count_same_letters = count_same_letters+1
                    print(count_same_letters)
                    print(len(wordb))
        return count_same_letters == len(wordb)
    else:
        return False

print(is_anagram("anagram","nagaram"))

while trying the string 'abc' abd 'bca' the output was True as I expected, but the strings 'anagram' and 'nagaram'returns False

Answer: You are going through each pair of letters.
For instance if you have two 5-letter words, you're going through 25 pairs of letters. If you input aaaab and baaaa, your count_same_letters counter will get to 4*4 + 1*1 (4*4 pairs of a's and 1*1 pairs of b's).
Change your algorithm.
Title: What does it mean if -1 is returned for .get_device() for torch tensor?
Tags: python,pytorch,pytorch-geometric

Question: I am using pytorch geometric to train a graph neural network. The problem that led to this question is the following error:

RuntimeError: Expected all tensors to be on the same device, but found
at least two devices, cpu and cuda:0! (when checking argument for
argument mat1 in method wrapper_addmm)

So, I am trying to check which device the tensors are loaded on, and when I run data.x.get_device() and data.edge_index.get_device(), I get -1 for each. What does -1 mean?
In general, I am a bit confused as to when I need to transfer data to the device (whether cpu or gpu), but I assume for each epoch, I simply use .to(device) on my tensors to add to the proper device (but as of now I am not using .to(device) since I am just testing with cpu).
Additional context:
I am running ubuntu 20, and I didn't see this issue until installing cuda (i.e., I was able to train/test the model on cpu but only having this issue after installing cuda and updating nvidia drivers).
I have cuda 11.7 installed on my system with an nvidia driver compatible up to cuda 12 (e.g., cuda 12 is listed with nvidia-smi), and the output of torch.version.cuda is 11.7. Regardless, I am simply trying to use the cpu at the moment, but will use the gpu once this device issue is resolved.

Answer: -1 means the tensors are on CPU. when you do .to(device), what is your device variable initialized as? If you want to use only CPU, I suggest initializing device as device = torch.device("cpu") and running your python code with CUDA_VISIBLE_DEVICES='' python your_code.py ...
Typically, if you are passing your tensors to a model, PyTorch expects your tensors to be on the same device as your model. If you are passing multiple tensors to a method, such as your loss function ``nn.CrossEntropy()```, PyTorch expects both your tensors (predictions and labels) to be on the same device.
Title: Pip is looking for python in wrong directory
Tags: python,directory,python-3.9

Question: Pip is looking for python in a directory that no longer exists. I downgraded from python 3.10 to python 3.9 using Microsoft app store after my original python 3.10 had an error running pip install pyqt5-tools. I have tried adding my new python to PATH but it doesn't work. I am trying to convert a UI file to a PY file using pyuic5.
I tried
C:\WINDOWS\system32>python --version

It gave
No Python at '"C:\Program Files\Python310\python.exe'

I also tried
where python

and it gave
C:\Users\Ponsar Kumzhi\PycharmProjects\pythonProject1\venv\Scripts\python.exe

This suggests that Windows knows where the correct installation of python is located.
I have tried adding this new directory to path and it doesn't work. What should I do?

Answer: You probably missed out something or there is some issues with your computer. To update the PATH environment variable is probably the only solution to fix this issue.

Search for "Environment Variables" in the search bar.
Click on "Edit the system environment variables".
In the System Properties window, click on the "Environment Variables" button.
Scroll down to the "Path" variable in the "System Variables" section and select it.
Click on the "Edit" button and "New" button and add the path to the Python 3.9 installation directory. For your case, it is: C:\Users\Ponsar Kumzhi\PycharmProjects\pythonProject1\venv\Scripts\python.exe.
Click on "OK" lastly.

After you finished the above mentioned steps, press Windows + R and type cmd, then press enter. Try typing "py --version" and it should print "Python [version]". You should then be able to use pip to install packages and use pyuic5 command.
Title: AttributeError: EagerTensor object has no attribute 'astype'
Tags: python,image,tensorflow,keras,tensor

Question: I am trying to do a GradCAM Heatmap in Google Colab like so:
import tensorflow as tf
from tensorflow.keras import backend as K
from tf_keras_vis.activation_maximization import ActivationMaximization
from tf_keras_vis.utils.callbacks import Print

def model_modifier(m):
    m.layers[-1].activation = tf.keras.activations.linear

activation_maximization = ActivationMaximization(model, model_modifier)

loss = lambda x: K.mean(x[:, 1])

activation = activation_maximization(loss, callbacks=[Print(interval=100)])
image = activation[0].astype(np.uint8)         # <----- error

f, ax = plt.subplots(figsize=(10, 5), subplot_kw={'xticks': [], 'yticks': []})
ax.imshow(image)
plt.show()

but I get an error
AttributeError: EagerTensor object has no attribute 'astype'.

Answer: Convert to numpy first with activation[0].numpy(), then you should be able to use numpy array methods.
Title: Numpy generating array from repeated function
Tags: python,numpy,bernoulli-probability

Question: I'm trying to generate an numpy array with randomly generated elements (it's similar like bernouilli distribution. The thing is I want to achieve it without using numpy.random.binomial function). Only function that fits after searching documentation and some forums is numpy.fromfunction. But I encountered a problem that this function doesn't generate n elements, but one.
I expect output something like:
[0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1]

This function generates 1 element only, no matter what shape is in tuple:
np.fromfunction(lambda i : np.random.choice([0, 1], p=[0.1, 0.9]), (20, ))

#returns 1 or 0
np.fromfunction(lambda i ,j : np.random.choice([0, 1], p=[0.1, 0.9]), (20, 1))

#still returns 1 or 0
Though I tried implementing "i" into the output stupidest way possible but.. it changed something, but still didn't help:
np.fromfunction(lambda i : i*0 + np.random.choice([0, 1], p=[0.1, 0.9]), (20, ))

#returns
array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

It's closer to the shape I want, but this ones or zeros are just repeated on all the array, so nothing's changed really.
To sum up: I have a function f() that generates randomly with some probability 0 and 1, and is there any other function in numpy that can repeat function f() on array, or maybe some way to repair the example above?

Answer: Sure, np.random.choice([0, 1], p=[0.1, 0.9], size=(20,)) or also there is np.random.randint(low=0, high=2, size=(20,)).
Title: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), Python
Tags: python,python-3.x,web-scraping,web,python-requests

Question: I made a telegram bot and deployed it on Render.
All works fine for like 3 hours then the bot gets down and throws this error:
`raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))`

Worth mentioning that the server keeps being Live despite the error, but the bot gets totally out of service.
And when re-deploying it, it works fine for another 3 hours and then bot dies but server keeps Live.
I tried adding this code:
time.sleep(0.45)
r = requests.get(get_url(name), headers=headers)
if r.status_code == 200:
soup = bs(r.content, features='html.parser')

But did not help that much.
Any solutions?

Answer: Turned out I just needed to use
bot.infinity_polling()
instead of bot.polling()
Title: What is the maximum size of a csv file?
Tags: python,excel,windows,csv,file

Question: I have 70 files in format with an actual file size of 800 MB each. Total original file size: 56000 MB.
Unzipping, moving, and processing (at least renaming the files) become quite challenging and time-consuming tasks in Windows i7.
I was considering if it will be a good idea to concatenate all the files in one file. So, one giant file of size of 56000 MB.
Is there any limitation on the maximum size a CSV file can have? Also, is it a good idea to work with a such big single file or shall I continue with 70 separate files?

Answer: The only limitation is your storage limit, in theory a csv can be of unlimited size.
But if it's to big it may be unusable for practical use because limitations of your read and write speed, memory size and so on
Title: Meaning of this error message python version conflicts
Tags: python,python-3.x,conda,environment

Question: While trying to install package: line-profiler-pycharm, I install it using pip inside my conda environment called wcs. But when I try to run it, I get the following error:
 22:01  Error running 'prep_network': Could not 
do Profile Lines execution because the
python package 'line-profile-pycharm' is not 
installed to Python 3.8 (wcs): Python 3.10.5 (/Users/myusername/opt/anaconda3/envs/wcs/bin/python)

Why does it show to python 3.8 and then python 3.10 after the colon :?
I checked the python versions inside my ˜/anaconda3/wcs/bin/python , I can see python 3.11 and python3.10, Python3.8 does not even exist in there.

Answer: Type $ which pip and compare with $ which python.
They likely come from different places.
To avoid the mismatch, prefer $ python -m pip install ...
Going forward, it would be best to add "line-profiler-pycharm"
to your project's environment.yml,
and then $ conda env update will properly install it for you.
If your config gets messed up, it is always harmless
to discard the environment (anaconda3/envs/wcs) and
use $ conda env update to re-create it from scratch.
Title: How can I update my Python version and keeping all the packages?
Tags: python,python-3.x

Question: I want to ask you a question about the Python update.
If I wanted to update my Python version from the 3.10 to 3.11, what should I do?
And, most important, do I have to reinstall all the packages like Pandas, Numpy, Matplotlib, ...?
In your opinion, when is needed to update it?

Answer: Unless there is some security threat or some particularly new fun-ctionality, I wouldn't update too much between minor versions.
Then thanks to pip managing python packages is easy :
You can use pip freeze contextualized to a version, like :
python3.10 -m pip freeze
This will output a text with all libraries and their versions used by this specific binary.
You can put that into a file :
python3.10 -m pip freeze > /tmp/python3.10.pip.txt
And then install it with another version :
python3.11 -m pip install -r /tmp/python3.10.pip.txt
Title: Confused by python async for loop--executes sequentially
Tags: python,python-asyncio

Question: I am new to asyncio and trying to understand basic for loop behavior. The code below executes sequentially, but my naive assumption was that while the sleeps are occurring, other items could be fetched via the for loop and start processing. But that doesn't seem to happen.
For example, while the code is "doing something else with 1" it seems like it could fetch the next item from the loop and start working on it while waiting for the sleep to end on item 1. But when I run, it executes sequentially with pauses for the sleeps like a non-async program.
What am I missing here?
import asyncio


class CustomIterator():
    def __init__(self):
        self.counter = 0

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.counter >= 3:
            raise StopAsyncIteration
        await asyncio.sleep(1)
        self.counter += 1
        return self.counter


async def f(item):
    print(f"doing something with {item}")
    await asyncio.sleep(3)


async def f2(item):
    print(f"doing something else with {item}")
    await asyncio.sleep(2)


async def do_async_stuff():
    async for item in CustomIterator():
        print(f"got {item}")
        await f(item)
        await f2(item)


if __name__ == '__main__':
    asyncio.run(do_async_stuff())

Output:
got 1
doing something with 1
doing something else with 1
got 2
doing something with 2
doing something else with 2
got 3
doing something with 3
doing something else with 3

Answer: I think you have a common misunderstanding of how async works.  You have written your program to be synchronous.  await foo() says to call foo(), and feel free to go do something else while we're waiting for foo to return with its answer.  Likewise, getting the next element from your custom iterator says "get the next element of this iterator, but feel free to go do something else while waiting for the result".  In both cases, you have nothing else to do, so your code wants.
If it is safe for two things in your code to run at once, it is your job to say so, using appropriate primitives.
Title: Confused by python async for loop--executes sequentially
Tags: python,python-asyncio

Question: I am new to asyncio and trying to understand basic for loop behavior. The code below executes sequentially, but my naive assumption was that while the sleeps are occurring, other items could be fetched via the for loop and start processing. But that doesn't seem to happen.
For example, while the code is "doing something else with 1" it seems like it could fetch the next item from the loop and start working on it while waiting for the sleep to end on item 1. But when I run, it executes sequentially with pauses for the sleeps like a non-async program.
What am I missing here?
import asyncio


class CustomIterator():
    def __init__(self):
        self.counter = 0

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.counter >= 3:
            raise StopAsyncIteration
        await asyncio.sleep(1)
        self.counter += 1
        return self.counter


async def f(item):
    print(f"doing something with {item}")
    await asyncio.sleep(3)


async def f2(item):
    print(f"doing something else with {item}")
    await asyncio.sleep(2)


async def do_async_stuff():
    async for item in CustomIterator():
        print(f"got {item}")
        await f(item)
        await f2(item)


if __name__ == '__main__':
    asyncio.run(do_async_stuff())

Output:
got 1
doing something with 1
doing something else with 1
got 2
doing something with 2
doing something else with 2
got 3
doing something with 3
doing something else with 3

Answer: To add more context on the other great answers, I'd like to detail a bit more in depth some definitions (quite simplified) which may help you to better understand the issue:

Concurrency refers to the ability to execute multiple tasks at seemingly the same period of time.
Parallelism refers to the ability to execute multiple tasks at physically the same period of time.
Asynchrony refers to actual ways to deal with concurrent tasks.

They are many of such ways of dealing with concurrent tasks at the application level, such as:

callback-based (callback functions),
dispatch queues,
the future and promise pattern,
the async/await pattern,
etc.

Using one of these forms of asynchronous programming does not guarantee that your program will execute concurrently, you often have to explicitly state that.
AsyncIO —as stated in the documentation— is "a library to write concurrent code using the async/await syntax.", and as its name suggests, is mainly dedicated to deal with I/O (Input/Output) asynchronous events (but not limited to).
To create a concurrent program using AsyncIO you need two things:

using the async/await syntax, and
explicitly schedule tasks concurrently, for instance with asyncio.gather and asyncio.create_task.

Without that, your program will generally executes sequentially as you observed in your example.
Title: Can i initialize optimizer before changing the last layer of my model
Tags: python,pytorch

Question: Say I want to change the last layer of my code but my optimizer is defined on the top of my scripts, what is a better practise?
batch_size = 8
learning_rate = 2e-4
num_epochs = 100

cnn = models.resnet18(weights='DEFAULT')
loss_func = nn.CrossEntropyLoss()
optimizer = optim.Adam(cnn.parameters(), lr=learning_rate)

def main(): 
  dataset= datasets.ImageFolder(root=mydir)
  loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
  num_classes = len(dataset.classes)
  num_ftrs = cnn.fc.in_features
  cnn.fc = nn.Linear(num_ftrs, num_classes)
  
  # Do i need to reinitialize the optimizer again here? 

if __name__ == '__main__':
  main()

Answer: No, the optimizer would not be aware of the weights for the new last layer unless you initialize or reinitilize the optimizer after modifying the model.
You should define the optimizer after creating the model and modifying any layers as needed.
Title: What happens when there are no await after create_task
Tags: python,python-3.x,python-asyncio

Question: I am trying to understand asyncio and I am quite confused about tasks.
Specifically, I know a task will run when the current coroutine is suspended and controll is passed to the event loop. What if await is never called after a task is created, and the coroutine just exit?
For example
import asyncio

async def print_stuff(s):
    print(s)

async def print_wrapper(s):
    print(f"{s}: create task")
    asyncio.create_task(print_stuff(s))
    print(f"{s}: sleeping")
    print(f"{s}: ending coroutine")

async def sleep_wrapper(s):
    print(f"{s}: before sleep")
    await asyncio.sleep(2)
    print(f"{s}: after sleep")

async def main():
    asyncio.create_task(print_stuff("stuff"))
    asyncio.create_task(print_wrapper("wrapper"))
    asyncio.create_task(sleep_wrapper("abc"))

asyncio.run(main())
print("after main")

The output is
stuff
wrapper: create task
wrapper: sleeping
wrapper: ending coroutine
abc: before sleep
after main

What I don't understand is

why would I see any output from the tasks at all, after main ends, shouldn't the event loop ends with it?
why "wrapper" and "abc: after sleep" were not printed while the other messages were?

Answer: I briefly looked at the asyncio source and I think the event loop makes one loop (or maybe two?) iterations and then it returns, because the stop condition (finished main) is true.
There are 4 tasks at the beginning, 3 created in the main and the 4th is the main coroutine itself. In one loop iteration each task that is ready is run one "step", i.e. new tasks from the beginning, other tasks from the point where the previous step finished. A "step" always finishes at the point where the task cannot continue and that is either the end of the corresponding coroutine (a return or after the last instruction or an exception) or at an await that cannot immediately return a value. In the first case, the task becomes "done". In the latter case the task becomes not ready and waits for some pending future value. When the future becomes "done", the associated task can be scheduled to run again. In your code it is the sleep where the "step" ends.
Title: How do I force the usual "default" formatting that pandas dataframes output to?
Tags: python,pandas,jupyter-notebook

Question: I'm using a jupyter like platform that has truly terrible visual capabilities and one of the annoying things is that when simply seeing a dataframe by having it be the last thing in a cell all the usual pandas formatting is gone if I turn it into a styler.
What I mean by that is:

alternating rows white / grey
hover over row gets me nothing instead of that blue highlighting
numbers are aligned in weird ways

I want to use styler because it allows me to format numbers easily (ie things like '{:,.0f}' or '{:.1%}'.
Is there a way to force the formatting to match the "default" in some easy way? eg where is the default stored?
df # this works fine as an output


df.style # this loses all default formatting

Answer: Some editors rely on CSS classes to apply their own browser specific styling.
At one time google colab refused to auto format a Styler since DataFrame HTML adds the dataframe class in <table class="dataframe">, whereas Styler doesnt.
Try using df.style.set_table_attributes('class="dataframe"')
Title: Append S3 File in Lambda, Python
Tags: python,amazon-web-services,amazon-s3,aws-lambda

Question: I understand it's not recommendable way but we need to update the same S3 file using Lambda.
We'd like to write and add the count result in the same file in S3.
Usually, it's counting and updating one by one in sequential order in the same job session. But we can't deny any possibility that the scheduled jobs using same Lambda function can run simultaneously, especially at the busy time. In such case, the file can be updated in unintended way or corrupsed in the worst senario.
I think we can set the concurrency 0 of that Lambda function and make it always run in sequential order, but I'm afraid it might cause the performance issue instaed.
If you have any better idea for concurrency prevention or any error countermeasure in Python code below, please advise me.
Thank you for your help in advance.
-- lambda_handler.py

import json
from s3 import *

def lambda_handler(event, context):
    s3_append('yyyymmdd/result.txt', 'yyyymmdd/result.txt', 'append str')


-- function S3.py    
import boto3
from datetime import datetime

bucket_name = 's3-sachiko-aws3-new2'
s3_resource = boto3.resource("s3")

# S3 File Append
def s3_append(target_path, append_path, append_str):

    target_context = s3_resource.Object(bucket_name, target_path).get()["Body"].read()  
    s3_resource.Object(bucket_name, append_path).put(Body=target_context + b"\n" +  bytes(append_str, 'utf-8'))

Answer: If I understand your requirement, you want to do two things:

Make a freeform update to an S3 file based on an upstream request
Make a simple calculated update to the same file based on the result of Update 1.

If this is the case, you can set up a SQS FIFO queue and push the requests from the upstream system into it.  Then you can set up an Event Source Mapping lambda that will pull freeform updates one-by-one from the queue, make Updates 1 and 2, write the file back to S3 and then pick up the next item in the queue.
Title: Bases must be type error when running "from google.cloud import bigquery" in jupyter notebook in GCP
Tags: python,google-bigquery

Question: I tried running the following:
from google.cloud import bigquery

But from some reason, I keep getting this "Bases must be types" error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-1035661e8528> in <module>
----> 1 from google.cloud import bigquery

/opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/__init__.py in <module>
     33 __version__ = bigquery_version.__version__
     34 
---> 35 from google.cloud.bigquery.client import Client
     36 from google.cloud.bigquery.dataset import AccessEntry
     37 from google.cloud.bigquery.dataset import Dataset

/opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/client.py in <module>
     58 
     59 import google.api_core.client_options
---> 60 import google.api_core.exceptions as core_exceptions
     61 from google.api_core.iam import Policy
     62 from google.api_core import page_iterator

/opt/conda/lib/python3.7/site-packages/google/api_core/exceptions.py in <module>
     27 import warnings
     28 
---> 29 from google.rpc import error_details_pb2
     30 
     31 try:

/opt/conda/lib/python3.7/site-packages/google/rpc/error_details_pb2.py in <module>
     18 # source: google/rpc/error_details.proto
     19 """Generated protocol buffer code."""
---> 20 from google.protobuf import descriptor as _descriptor
     21 from google.protobuf import descriptor_pool as _descriptor_pool
     22 from google.protobuf import message as _message

/opt/conda/lib/python3.7/site-packages/google/protobuf/descriptor.py in <module>
     38 import warnings
     39 
---> 40 from google.protobuf.internal import api_implementation
     41 
     42 _USE_C_DESCRIPTORS = False

/opt/conda/lib/python3.7/site-packages/google/protobuf/internal/api_implementation.py in <module>
    102   try:
    103     # pylint: disable=g-import-not-at-top
--> 104     from google.protobuf.pyext import _message
    105     sys.modules['google3.net.proto2.python.internal.cpp._message'] = _message
    106     _c_module = _message

TypeError: bases must be types

I'm not sure what I did wrong. Maybe I install or uninstalled a package somewhere. But now I can't get this simple command to work. Any help would be greatly appreciated.

Answer: correct input in terminal to change protobuf version (my acct. was unable to comment on the above solution so had to add as answer instead...)

pip install --upgrade protobuf==3.20.1
Title: Bases must be type error when running "from google.cloud import bigquery" in jupyter notebook in GCP
Tags: python,google-bigquery

Question: I tried running the following:
from google.cloud import bigquery

But from some reason, I keep getting this "Bases must be types" error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-1035661e8528> in <module>
----> 1 from google.cloud import bigquery

/opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/__init__.py in <module>
     33 __version__ = bigquery_version.__version__
     34 
---> 35 from google.cloud.bigquery.client import Client
     36 from google.cloud.bigquery.dataset import AccessEntry
     37 from google.cloud.bigquery.dataset import Dataset

/opt/conda/lib/python3.7/site-packages/google/cloud/bigquery/client.py in <module>
     58 
     59 import google.api_core.client_options
---> 60 import google.api_core.exceptions as core_exceptions
     61 from google.api_core.iam import Policy
     62 from google.api_core import page_iterator

/opt/conda/lib/python3.7/site-packages/google/api_core/exceptions.py in <module>
     27 import warnings
     28 
---> 29 from google.rpc import error_details_pb2
     30 
     31 try:

/opt/conda/lib/python3.7/site-packages/google/rpc/error_details_pb2.py in <module>
     18 # source: google/rpc/error_details.proto
     19 """Generated protocol buffer code."""
---> 20 from google.protobuf import descriptor as _descriptor
     21 from google.protobuf import descriptor_pool as _descriptor_pool
     22 from google.protobuf import message as _message

/opt/conda/lib/python3.7/site-packages/google/protobuf/descriptor.py in <module>
     38 import warnings
     39 
---> 40 from google.protobuf.internal import api_implementation
     41 
     42 _USE_C_DESCRIPTORS = False

/opt/conda/lib/python3.7/site-packages/google/protobuf/internal/api_implementation.py in <module>
    102   try:
    103     # pylint: disable=g-import-not-at-top
--> 104     from google.protobuf.pyext import _message
    105     sys.modules['google3.net.proto2.python.internal.cpp._message'] = _message
    106     _c_module = _message

TypeError: bases must be types

I'm not sure what I did wrong. Maybe I install or uninstalled a package somewhere. But now I can't get this simple command to work. Any help would be greatly appreciated.

Answer: Marking this as an answer for better visibility, as this resolved the question: pip installing protobuff v3.20.1 as @runner16 mentioned

pip install protobuf == 3.20.1
Title: Pandas dataframe - transform selected cell values based on their suffix
Tags: python,pandas,dataframe,type-conversion

Question: I have a dataframe as below:
data_dict = {'id': {0: 'G1', 1: 'G2', 2: 'G3'},
 'S': {0: 35.74, 1: 36.84, 2: 38.37},
 'A': {0: 8.34, 1: '2.83%', 2: 10.55},
 'C': {0: '6.63%', 1: '5.29%', 2: 3.6}}
df = pd.DataFrame(data_dict) 

I want to multiply all the values in the data frame with 10000 (except under the column 'id' - 1st column) if they endswith %:
cols = df.columns[1:]
for index, row in df.loc[:, df.columns != 'id'].iterrows():
    for c in cols:
        if str(row[c]).endswith('%'):
            data_value = str(row[c])
            data_value = data_value.replace('%',"")
            df.at[index,c]= float(data_value) * 10000

Finally, this sets all the columns values (except the first column) to numeric:
df[cols[1:]] = df[cols[1:]].apply(pd.to_numeric, errors='coerce')

Is there a simple way to convert the values instead of iterating the rows?

Answer: You rewrite the code by using applymap.