Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: How To Iterate Through a Collection of Images with Navigation Buttons Using Flask
Tags: python,html,flask

Question: I am learning how to program and am working on my final project for my intro class.
I am building a website using the Flask framework. The purpose of the website is to allow the user to upload images and display them to the public. I store those images in a folder. I use Bootstrap and put the images each on a card. Since there may be many images, I want to be able to show a few at a time and allow the user to click forward and backward through the set.
I am able to loop through the whole list successfully. I have also found a way to show only 6 at a time.
I have a global variable STEP which is basically a counter that starts at 0.
I have a global variable INTERVAL which defines how many images to load each time (6).
"rows" is the SQL query that I pass to the page with info about each image.
    <div class="container">
        <div class="row">
            {% for row in rows[step:(step+interval)] %}
            <div class="col-md-4 p-3">
                <div class="card-deck h-100">
                    <div class="card text-center" style="width: 18rem;">
                    <img src="../static/uploads_Andrea/{{ row.filename }}" class="card-img-top" alt="...">
                    <div class="card-body">
                        <h4 class="card-title">{{ row.title }}</h4>
                        <h6 class="card-title"><i>{{ row.date }}</i></h6>
                        <p class="card-text" text-align:center;>{{ row.description }}</p>
                    </div>
                    </div>
                </div>
            </div>

            {% endfor %}

        </div>

I want to display after each set of six images two buttons: "older" and "newer" (the images have dates associated and are displayed in reverse chronological order).
I am asking for some help in understanding how to proceed from here. I wanted to just have Flask recognize which button is clicked and then either add or subtract INTERVAL from the current STEP and run the loop again. However, I can't find anything suggesting this capability in Flask.
I suspect that I need to create a form that sends the button click back to the app so that the app can add or subtract INTERVAL from the current STEP and refresh the page and run the loop. But I cannot find anything that suggests how a button can initiate this action.
Any help or direction would be appreciated. I have seen some answers get close but appear to be much more complex than what I am looking for.
I tried to have a button action that would pass back to the app the need to add/subtract interval from step and re-POST the page but really did not know how to go about this.

Answer: Please

Update your Flask code to accept optional GET parameters for starting_record and page_size, which would then also be used in your SQL query
Use the template tool (e.g., Jinja) to create links for before and after pages using something like <a href="loadimages?starting_record=$(starting_record - page_size)&page_size=$(page_size)">previous page</a>

Hope this helps.
Title: File path error while converting .md to .pdf
Tags: python,subprocess,pandoc

Question: In a python script I'm trying to convert a .md file to .pdf using subprocess and pandoc module as given below
subprocess.run(["pandoc", "--standalone", "--pdf-engine=xelatex", "/file/path/file.md", "-o", "/file/path/file.pdf")

However I get the following error with image files in the markdown file
[WARNING] Could not fetch resource 'image.png': replacing image with description

The image file is in the same directory as the .md file, and the same image path works fine when it is converted to .html file. Can someone point out what the problem is here?

Answer: Wild quess; when you run your python script calling pandoc, pandoc gets executed on some other directory where the images are located. Maybe pass --resource-path that points to the same directory where your markdown file is or point cwd kwarg of subprocess.run to the same directory where the markdown file resides.
Title: In network programming, must the accepting end close both sockets?
Tags: python,c,sockets

Question: I raised a question earlier, and one of the answers involved another question.
That is to say, in general, the accepting end should create two sockets to establish the accepting end. like this:
Python
from socket import *
sockfd = socket(...)
# ...
client_sockfd, addr = sockfd.accept()
# ...
client_sockfd.close()
sockfd.close()

C
int sockfd, client_sockfd;
sockfd = socket(...);
// ...
client_sockfd = accept(sockfd, ...);
// ...
shutdown(client_sockfd, 2);
shutdown(sockfd, 2);
close(client_sockfd);
close(sockfd);


So can we skip the task of creating the client_sockfd variable? like this:
Python
sockfd = socket(...)
# ...
sockfd, addr = sockfd.accept()
# ...
sockfd.close()

C
int sockfd;
struct sockaddr_in server, client;
socklen_t client_size = sizeof(client);
sockfd = socket(...);
// ...
sockfd = accept(sockfd, (struct sockaddr *)&client, &client_size);

Or it could be like this:
Python
sockfd = socket(...)
# ...
client_sockfd, addr = sockfd.accept()
sockfd.close()
# ...
client_sockfd.close()

C
int sockfd = socket(...);
int client_sockfd;
// ...
client_sockfd = accept(sockfd, ...);
shutdown(sockfd, 2);
close(sockfd);
// ...
shutdown(client_sockfd, 2);
close(client_sockfd);

As shown in the above code, can we use only one socket to complete the accepting end of the entire network programming? Is there any problem with this? (At least I didn't have any problems writing the program like this myself)

Answer: I'm not sure if there is a "problem", but it seems relevant to explain whats going on here:
When you call sockfd.accept() it blocks until a connection comes in. When one does come in it returns another socket which "represents" the connection, and can be used to send and receive data on the connection. The reason for the multiple calls to socket.close() is that in your first example client_sockfd.close() only closes the connection. You can still call sockfd.accept() again to receive another connection, and many socket servers use socket.accept() in a while loop. Calling sockfd.close() will close sockfd, usually when no further connections (calls to sockfd.accept()) need to be made.
Title: In network programming, must the accepting end close both sockets?
Tags: python,c,sockets

Question: I raised a question earlier, and one of the answers involved another question.
That is to say, in general, the accepting end should create two sockets to establish the accepting end. like this:
Python
from socket import *
sockfd = socket(...)
# ...
client_sockfd, addr = sockfd.accept()
# ...
client_sockfd.close()
sockfd.close()

C
int sockfd, client_sockfd;
sockfd = socket(...);
// ...
client_sockfd = accept(sockfd, ...);
// ...
shutdown(client_sockfd, 2);
shutdown(sockfd, 2);
close(client_sockfd);
close(sockfd);


So can we skip the task of creating the client_sockfd variable? like this:
Python
sockfd = socket(...)
# ...
sockfd, addr = sockfd.accept()
# ...
sockfd.close()

C
int sockfd;
struct sockaddr_in server, client;
socklen_t client_size = sizeof(client);
sockfd = socket(...);
// ...
sockfd = accept(sockfd, (struct sockaddr *)&client, &client_size);

Or it could be like this:
Python
sockfd = socket(...)
# ...
client_sockfd, addr = sockfd.accept()
sockfd.close()
# ...
client_sockfd.close()

C
int sockfd = socket(...);
int client_sockfd;
// ...
client_sockfd = accept(sockfd, ...);
shutdown(sockfd, 2);
close(sockfd);
// ...
shutdown(client_sockfd, 2);
close(client_sockfd);

As shown in the above code, can we use only one socket to complete the accepting end of the entire network programming? Is there any problem with this? (At least I didn't have any problems writing the program like this myself)

Answer: You are always creating two sockets: the server socket and the first accepted client socket.  When you close them is up to you.
Using the same name for both in Python will still close the server socket (eventually) because the object reference count goes to zero, but in C you leak the socket.
Title: scipy.interpolate.BSpline: Are there rules of thumb for the length of the enlarges x vector?
Tags: python,scipy,spline

Question: I have code working to smooth some accelerometer x,y,z values in columns (toy data):
import numpy as np
import pandas as pd
from scipy.interpolate import make_interp_spline, BSpline
idx = [1,2,3,4]
ts_length = 200

idx = np.array(idx,dtype=int)
accel = [[-0.7437,0.1118,-0.5367],
         [-0.5471,0.0062,-0.6338],
         [-0.7437,0.1216,-0.5255],
         [-0.4437,0.3216,-0.3255],
]
print(accel)
accel = np.array(accel,dtype=float)
accel_sm = np.zeros([ts_length,3])
idxnew = np.linspace(idx.min(), idx.max(), ts_length) 
for i in range (0,3):
         c = accel[:,i]
         spl = make_interp_spline(idx, c, k=3)
         c_smooth = spl(idxnew)
         accel_sm[:, i] = c_smooth
print(accel_sm)

I just picked ts_length = 200 out of a hat since I was only trying to get the example working. But I was wondering: Are there rules of thumb for how big it should be relative to len(idx) to do a good job of smoothing without making the smoothed array unnecessarily large?

Answer: make_interp_spline does not do any smoothing, its result passes through the data.
For smoothing, use make_smoothing_spline or splrep.
Title: Stop Prometheus client in python script
Tags: python,prometheus

Question: Need to stop prometheus_client.http_server with Python code.
I have external Python application that is to be unit-tested. It has method that starts prometheus_client with start_http_server(port). I need to unit-test it with several tests (indeed I don't test prometheus but other functionalities but can't take prometheus out of the code). As I need several tests so I need to start-stop method several times, but it exits after first attempt as Prometheus client is holding port and can't start again on the same port (OSError: [Errno 48] Address already in use).
The question: is there any way to stop prometheus_client.http_server with Python? Didn't find anything in prometheus_client docs about shutting it down...

Answer: Well, it's not exact answer to my question but anyway: I decided to start_http_server = Mock(return_value=None) so that it's not starting in unit-tests.
Title: "The model 'MPTForCausalLM' is not supported for text-generation"- The following warning is coming when trying to use MPT-7B instruct
Tags: python-3.x,deep-learning,nlp

Question: I am using a VM of GCP(e2-highmem-4 (Efficient Instance, 4 vCPUs, 32 GB RAM)) to load the model and use it. Here is the code I have written-
import torch
from transformers import pipeline
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import transformers
config = transformers.AutoConfig.from_pretrained(
  'mosaicml/mpt-7b-instruct',
  trust_remote_code=True,
)
# config.attn_config['attn_impl'] = 'flash'

model = transformers.AutoModelForCausalLM.from_pretrained(
  'mosaicml/mpt-7b-instruct',
  config=config,
  torch_dtype=torch.bfloat16,
  trust_remote_code=True,
  cache_dir="./cache"
)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b", cache_dir="./cache")
text_gen = pipeline("text-generation", model=model, tokenizer=tokenizer)
text_gen(text_inputs="what is 2+2?")

Now the code is taking way too long to generate the text. Am I doing something wrong? or is there any way to make things faster?
Also, when creating the pipeline, I am getting the following warning-\
The model 'MPTForCausalLM' is not supported for text-generation
I tried generating text by using it but it was stuck for a long time.

Answer: You might want to try a GPU instances, because trying to use bigger LLMS like this with CPUs is pretty much a lost cause now.
Anyhow, I also got that "The model 'MPTForCausalLM' is not supported for text-generation" Which is why I ended up in this thread. Text Generation did work for me despite the warning.
Title: creating an excel vlookup in python script
Tags: python,excel,boto3,openpyxl,vlookup

Question: i'm creating an excel spreadsheet with data from my aws account. That's working fine, but i'm trying to insert a vlookup via the script. the formula gets written to the cell but excel doesnt evaluate it. Excel is set to calculate formulas automatically. When I open the file if I activate the formula bar and click the green tick the formula runs and I get the expected result. I'm wondering if i'm missing anything in my python code. Also using openpyxl to create the workbook. Any help would be really appreciated!
for row in range(2, wrkbook.active.max_row + 1):
    wrkbook.active[f'B{row}'].value = f'=VLOOKUP(A{row},Table1,2,FALSE)'

When I open the xlsx file, #NAME error is displayed.
Contents of formula bar: =VLOOKUP(A2,Table1,2,FALSE)
When I open the file if I activate the formula bar and click the green tick the formula runs and I get the expected result.

Answer: Openpyxl edits the xml files that make up an Excel xlsx file. It cannot calculate formulas or influence how or what Excel then does with the formulas when it opens the file. Therefore if the result when opening the file is the formulas show #NAME? there is nothing Openpyxl can do about that.
You have the option to refresh the cells when the workbook/worksheet is open. 
Again Openpyxl cannot do anything here since it has no interaction with Excel. It cannot tell Excel to refresh the range on open.
However you can open the workbook using a module that utilises Excel and refresh the cells and that will fix your issue. That is something a module like Xlwings could do for you. It should be noted at the same time that Xlwings can also add the data, create the table and enter the vlookup formulas. If you completed these steps in Xlwings (or at least used it to enter the formulas), when you open the workbook in Excel the formulas should be calculated and so refreshing is not required, i.e. if you were to use Xlwings to refresh you may as well use it to write the all data and avoid the issue in the first place.
Title: AttributeError: module 'altair' has no attribute 'selection_point'
Tags: python,google-colaboratory,altair,vega-lite

Question: I'm trying to make a drop down menu and I am getting this error below. I'm not really understanding why since I've used alt.Chart and alt.binding_select before and it didn't get any problems. I'm a little familiar with vegalite, and I'm trying out vega altair in colab.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-41-f38e2e2742ea> in <cell line: 1>()
----> 1 source1_select = alt.selection_point(fields=data.experience_level, bind=source1_drop)

AttributeError: module 'altair' has no attribute 'selection_point'

My code so far:
import altair as alt
import pandas as pd
data = pd.read_csv("ds_salaries.csv")

# wanted to add drop down menu to this 
graph1 = alt.Chart(data).mark_bar().encode( 
    x='count(job_title)',
    y='job_title'
)

source1_drop = alt.binding_select(options=['EN', 'SE'], name="Experience Level (Source 1)")
source2_drop = alt.binding_select(options=['EN', 'SE'], name="Experience Level (Source 2)")

# Error here
source1_select = alt.selection_point(fields=data.experience_level, bind=source1_drop)

alt.__file__
# '/usr/local/lib/python3.10/dist-packages/altair/__init__.py'
alt.__path__
# ['/usr/local/lib/python3.10/dist-packages/altair']


I'm not sure why I'm getting this error, since I followed the documentation. Any help would be greatly appreciated!

Answer: I solved it. Turns out I was looking at v5 documentation whilst colab was running v4
Title: New Discord Username System
Tags: python,discord,discord.py,bots

Question: so currently i have this python bot that when a user tries to store something the author name along with the discriminator('#') is stored as well
return "**Hello** **" + message.author.name + "#" + message.author.discriminator + "** ** :wave: "
so my question is when they remove the discriminator
and apply the new username system, how can i change that in my bot that it stores the username? also am i going to need to upgrade my discord python package and will they add something like this?
message.author.username
because when i currently try to call message.author.username i get this
'User' object has no attribute 'username`

Answer: The Discord.py package will be updated sometime after discord applies the username system.
For now, you can't use the new system as the developers of Discord.py need to see how the new system works before they implement changes.
As for your question:
You will have to update the Discord.py package when an update eventually comes out, otherwise you will not be able to use the updated methods. We also don't know how storing the username will be executed, but message.author.username seems like a good guess of what the method will be like
Title: How do i add memory to RetrievalQA.from_chain_type? or, how do I add a custom prompt to ConversationalRetrievalChain?
Tags: python,openai-api,chatgpt-api,langchain,py-langchain

Question: How do i add memory to RetrievalQA.from_chain_type? or, how do I add a custom prompt to ConversationalRetrievalChain?
For the past 2 weeks ive been trying to make a chatbot that can chat over documents (so not in just a semantic search/qa so with memory) but also with a custom prompt. I've tried every combination of all the chains and so far the closest I've gotten is ConversationalRetrievalChain, but without custom prompts, and RetrievalQA.from_chain_type but without memory

Answer: When using the ConversationBufferMemory I am using a very simple test to confirm whether memory is working on my chatbot, which is asking the chatbot "What was the first question I asked".
I always seems to get the same incorrect answer:
I'm sorry, but I don't have access to your initial question as I am an AI language model and I don't have the capability to track previous interactions. Could you please repeat your question?
Apart from this though it does seem that memory is working to an extent, for example if I ask non-contextual questions, it does seem to be able to answer correctly.
Has anyone else encountered this anomaly.
I'm not sure if I need to work on the prompting or why else this anomaly is arising.
Title: PyCharm runs a flask app but fails to debug it in python3.11
Tags: python,flask,pycharm

Question: I have multiple python versions on my machine (3.8, 3.9, 3.10 and 3.11) used with different projects. All versions run fine with PyCharm 2023.1.1 except 3.11.
I have a flask-based project which uses 3.11 and it runs fine. Nevertheless, when I try to debug it, the server starts and then throws the following error:
Connected to pydev debugger (build 231.8770.66)
*Serving Flask app 'app'
Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on https://127.0.0.1:5001
Press CTRL+C to quit
Restarting with stat
C:\Users\SomeUser\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe: can't open file 'C:\\Program': [Errno 2] No such file or directory
Process finished with exit code 2

The virtual environment was created by the PyCharm interpreter automatically and it is using python3.11. It also seems that python.exe tries to open a nonexistent folder called Program which I assume is Program Files, but I do not get why.
I tried changing/adding PATHs and PYTHONPATHs. Played with various configuration settings. Installed-reinstalled both python3.11 and PyCharm and so far nothing seems to work.
Any suggestions on what might be causing the issue, before I try an old version of PyCharm?
I tried changing environment variables for python3.11. I tried installing and reinstalling both python3.11 and PyCharm. I tried changing the settings. I enabled the g-event compatibility for the Python Debugger in Pycharm. What I did not try is using an older PyCharm version.

Answer: I tried removing all empty spaces in the path string to PyCharm and this fixes the issue. That is, if I install it in a custom folder for example C:/PyCharm
and rename the PyCharm autogenerated folder PyCharm 2023 to PyCharm_2023 it also works.
Title: Always getting RateLimitError in Pinecone
Tags: python,google-colaboratory,openai-api,langchain,vector-database

Question: I have been trying for a long time. I have shifted to smaller documents (thinking that I was feeding way too many documents) and now I am just trying to embed 60 documents into my Pinecone index.
docsearch = Pinecone.from_texts([t.page_content for t in texts], embeddings, index_name=index_name

This is not working. What else should I try to bypass the rate limit error?

Answer: If you have the free plan of OpenAI then it only allows 3 requests per minute.
You need to update to Pay as you go, which allows way more requests per minute.
Note that it's not related to ChatGPT plan. It's about the API plan.
Title: How to capture logs from native python logging module in gunicorn?
Tags: python,flask,gunicorn,python-logging

Question: I have a python flask/gunicorn project which includes the standard flask logging code. However, some of my code may not run in an application context (it has its own unit tests, including some fairly complicated functions in other files). These files use the native python logging mechanism. How do I capture those logs and write them to the same log file as the gunicorn/flask logs?
from flask import Flask, jsonify

app = Flask(__name__)

@app.route("/")
def index():
     app.logger.info("index hit")
     return jsonify("ok")

And I know the trick to capture the logging output and make it write to the gunicorn log:
if __name__ != "__main__":
    gunicorn_logger = logging.getLogger("gunicorn.error")
    app.logger.handlers = gunicorn_logger.handlers
    app.logger.setLevel(gunicorn_logger.level)
    app.logger.info("Gunicorn logging enabled")

However, I have other code which may not run within an application context. For example, it's tested in a unit test.
import logging
logger = logging.getLogger(__name__)

def my_external_function(*args):
   logger.info("My function has been called")
   # do something

When I invoke gunicorn in the usual way:
gunicorn app:app -b 0.0.0.0:8080 \
    --access-logfile /var/log/myapp/access.log \
    --error-logfile /var/log/myapp/error.log \
    --log-level INFO

Everything that starts with app.logger. will write to error.log while the code using the native python logging (logger... or logging.) will write to stdout.

Answer: Confirm that the app logger is in a different context by inspecting logging.root.manager.loggerDict. If so, separate processes writing to the same file sounds problematic.
It maybe be suitable to merge separate files in a viewer. Configure your logger to have a file handler and have the same formatting as the Flask logger. Get a tool like LogMx that has the option of merging two logs into the same pane, interleaving according to timestamp.
Title: How to resolve AttributeError: 'version_info' object has no attribute '__version__'
Tags: matplotlib,scipy,python-imaging-library,pyparsing,carla

Question: C:\Coursera\CarlaSimulator\PythonClient\Course1FinalProject>python module_7.py
Traceback (most recent call last):
  File "module_7.py", line 26, in <module>
    import matplotlib.pyplot as plt
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\matplotlib\__init__.py", line 131, in <module>
    from matplotlib.rcsetup import defaultParams, validate_backend, cycler
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\matplotlib\rcsetup.py", line 29, in <module>
    from matplotlib.fontconfig_pattern import parse_fontconfig_pattern
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\matplotlib\fontconfig_pattern.py", line 22, in <module>
    from pyparsing import (Literal, ZeroOrMore, Optional, Regex, StringEnd,
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\site-packages\pyparsing\__init__.py", line 130, in <module>
    __version__ = __version_info__.__version__
AttributeError: 'version_info' object has no attribute '__version__'

I was trying to connect the python file for controller of an autonomous car to Carlasimulator. I guess what I am doing is not that necessary. However when I try to call the python file it show like this.
Pillow>=3.1.2
numpy>=1.14.5
protobuf>=3.6.0
pygame>=1.9.4
matplotlib>=2.2.2
future>=0.16.0
scipy>=0.17.0

These are the requierments for the simulator. Other than Pillow and Scipy I could downgrade the versions of other dependencies. But the version of Pillow and Scipy are newer. However I am still getting the error, seems like the versions are not the problem.
Please help me

Answer: I could run the file just installing older version of matplotlib. I am a beginner in this field, but I guess it's with the compatibility of Carla and version of matplotlib. when I installed matplotlib 2.2.4 everything works fine. hope this would be useful.
Title: How to write a list comprehension instead of a nested loop In Python?
Tags: python,loops,list-comprehension

Question: I was dealing with different dataframes in python. The first dataframe named cities looks like this:
                         city  Population              NBA
0               New York City    20153634      Knicks Nets
1                 Los Angeles    13310447  Lakers Clippers
2      San Francisco Bay Area     6657982         Warriors
3                     Chicago     9512999    Bulls[note 9]
4           Dallas–Fort Worth     7233323        Mavericks

And second dataframe named nba_df look like this:
                  team   W   L   W/L%    GB   PS/G   PA/G   SRS  year League
0      Toronto Raptors  59  23  0.720   0.0  111.7  103.9  7.29  2018    NBA
1       Boston Celtics  55  27  0.671   4.0  104.0  100.4  3.23  2018    NBA
2   Philadelphia 76ers  52  30  0.634   7.0  109.8  105.3  4.30  2018    NBA
3  Cleveland Cavaliers  50  32  0.610   9.0  110.9  109.9  0.59  2018    NBA
4       Indiana Pacers  48  34  0.585  11.0  105.6  104.2  1.18  2018    NBA

What I am trying to do is to determine where the team is located from teams name. For ex. Philadelphia 76ers are located at Philadelpia. So to do that I wrote a nested loop below:
city = []
for i in nba_df['team']:
    added = False
    for j in i.split():
        for k in cities['city']:
            for l in k.split():
                if j == l and added == False:
                    city.append(k)
                    added = True
    if added == False:
        city.append('Not in list')
    
nba_df['city'] = city

The loop iterates through the nba_df['team'] and splits the string and iterates for each word over the cities['city'] column which are splitted too, and if a word matches with the other, then the city name will be added into list. Otherwise it will be added as Not in list. I know it has some bugs, what will happen if New York City is being iterated before just saying as an example New Orleans? Than it will append the New York City to the list and skip the rest of it. So it might append wrong cities. Anyways code does what I want but I wanted to write it in list comprehension. Because it looks amateur to me.
I tried something like this:
city = [k if j == l for l in k.split() in [k for k in cities['city'] in [j for j in i.split() in [i for i in nba_df['team']]]]]

But of course it didn't work. It raised and sytanx error pointing on l for l which is at the beginning.
What I am asking for you is to review my loop and I wonder how would you rewrite it? List comprehension or another way. How would you approach to the problem?
Thank you.

Answer: You forgot the braces at the first split() in the list comprehention.
Title: Issue in the requirements.txt file when trying to deploy to Vercel
Tags: python,django,vercel,railway

Question: I'm trying to deploy my Django web app, however, vercel is giving me this error when it failed to deploy:
Failed to run "pip3.9 install --disable-pip-version-check --target . --upgrade -r /vercel/path0/workout_log/requirements.txt"
Error: Command failed: pip3.9 install --disable-pip-version-check --target . --upgrade -r /vercel/path0/workout_log/requirements.txt
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      /tmp/pip-build-env-q1jqann4/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`

Here is my requirments.txt file:
asgiref==3.6.0
beautifulsoup4==4.12.0
certifi==2022.12.7
charset-normalizer==3.1.0
dj-database-url==1.2.0
Django==4.1.4
django-bootstrap4==22.3
django-environ==0.10.0
django-heroku==0.3.1
gunicorn==20.1.0
heroku==0.1.4
idna==3.4
psycopg2==2.9.5
psycopg2-binary==2.9.6
python-dateutil==1.5
requests==2.28.2
soupsieve==2.4
sqlparse==0.4.3
tzdata==2022.7
urllib3==1.26.15
whitenoise==6.4.0

I tried to use Python 3.9 to fix the problem. I also tried to run this command on my local environment, but it said that pip3.9 is not a recognized command.

Answer: I don't know if this will fix it, but I believe it is supposed to be psycopg2-binary~=2.9.6 and I don't know if psycopg2==2.9.5 is necessary. Try that and see if it works.
Title: Does async requests with limited number of concurrent requests generally run faster?
Tags: python,python-asyncio

Question: I was playing with async python code trying to improve its performance, and noticed that when I set a limit on number of simultaneously executing tasks via Semaphore, the code usually runs faster than if I don't set any limit and just allow it to make as many requests as it likes. I also noticed, that when I limit number of web requests I get connection errors less often. Is this a general case?

Answer: async functions let us run several tasks in parallel at the same time, but they don't spawn any new threads or new processes. They all run in the same, single thread of the Python interpreter. That is to say, the interpreter can only run as fast as a single core of your processor, and the tasks can only run as fast as the interpreter. (This is the meaning of "concurrent", as distinct from "parallel", where multiple physical processors are in use at the same time.)
That's why folks say that async is good for I/O, and nothing else. It doesn't add computation power, it only lets us do a lot of otherwise-blocking stuff like HTTP requests in parallel. It "recycles" that blocking time, which would otherwise just be idling on the CPU waiting for the network to respond, wasting that CPU time.
So by "recycling" the CPU time that would otherwise be wasted, more tasks increases requests/second, but only up to a point. Eventually, if you spawn too many tasks, then the interpreter and the processor it's running on spend more CPU time managing the tasks than actually waiting for some network response. Not to mention that remote servers have their own bottlenecks (nevermind anti-DDOS protection).
So, async doesn't change the fact that you only have so much speed in your single thread. Too many tasks will clog the interpreter, and too many requests to remote servers will cause them to get fussy with you (whether by bottleneck or by anti-DDOS measures). So yes, you do need to limit your maximum concurrency.
In my local tests with trio and httpx, I get around 400-500 requests/second with around 128-256 concurrent tasks. Using more tasks than that reduces my total requests/second while burning more CPU time -- the interpreter does more task-management than requesting, at that point.
(I could probably optimize my local code, save some task-management CPU time for my interpreter, and maybe if I did that I could get 600 or 800 requests/second, but frankly the remote server I talk to can't really handle more requests anyways, so optimizing my local CPU usage would be mostly a waste of time. Better to add new features instead.)
Title: How to set the facecolor of a plot for saved figures
Tags: python,matplotlib

Question: I am working with some code that uses matplotlib v.3.1.2 on Docker (I can't change this), and I can't figure out how to set the background color of my saved plots to a different color than white (while keeping the fig background white).
Looking for a solution, I found three different approaches -- but none of them work.
Method 1 (changes the background color of both fig and axis to azure):
import matplotlib.pyplot as plt

fig, axis = plt.subplots(nrows=2, ncols=1, facecolor='azure')
for ax in axis:
    ...

...

plt.savefig(..., facecolor=fig.get_facecolor(), transparent=True)

Method 2 (doesn't do anything, i.e., the background color of both fig and axis remains white):
import matplotlib.pyplot as plt

fig, axis = plt.subplots(nrows=2, ncols=1)
for ax in axis:
    ...
    ax.set_facecolor('azure')

...

plt.savefig(..., facecolor=fig.get_facecolor(), transparent=True)

Method 3 (same result as Method 2):
import matplotlib.pyplot as plt
plt.rcParams['axes.facecolor'] = 'azure'

fig, axis = plt.subplots(nrows=2, ncols=1)
for ax in axis:
    ...

...

plt.savefig(..., facecolor=fig.get_facecolor(), transparent=True)

What am I doing wrong?
Here is the complete test example
import os
import random
import matplotlib.pyplot as plt
import numpy as np

plt.rcParams['font.serif'] = 'Rockwell'
plt.rcParams['font.family'] = 'serif'

BACKGROUND_COLOR = (1, 1, 1)
FONT_COLOR = (0.1, 0.1, 0.1)
MY_PATH = ...

fig, axes = plt.subplots(nrows=2, ncols=1)

vals = [[random.uniform(1.0, 8.0) for i in range(10)], 
        [random.uniform(1.0, 8.0) for i in range(10)]]
x = [i for i in range(10)]

colors = ['#F15854', '#B276B2']
legends = ['Feature 1', 'Feature 2']
for (colr, leg, y) in zip(colors, legends, vals):
    for i in [0, 1]:
        axes[i].plot(x, y, label=leg, linewidth=2, color=colr, alpha=1)

frm_min = int(x[0])
frm_max = int(x[-1])
for ax in axes:
    range_x = np.arange(frm_min, frm_max + 1, 2)
    ax.set_xticks(range_x)
    ax.set_xticklabels(range_x, fontsize=10)
    range_y = range(0, 8, 1)
    ax.set_yticks(range_y)
    ax.set_yticklabels(range_y, fontsize=10)
    ax.set_xlim(frm_min, frm_max)
    ax.grid(which='major', axis='x', linestyle=':')
    ax.set_xlabel('Time (s)')
    ax.set_ylabel('Value')

    main_legend = ax.legend(loc=7, ncol=1, borderaxespad=-10.0, fontsize=16)
    main_frame = main_legend.get_frame()
    main_frame.set_facecolor(BACKGROUND_COLOR)
    main_frame.set_edgecolor(BACKGROUND_COLOR)
    for text in main_legend.get_texts():
        text.set_color(FONT_COLOR)

    ax.set_facecolor('azure')

fig.suptitle('Figure title', fontsize=24, ha='center', color=FONT_COLOR)
fig.tight_layout(rect=[0, 0, 0.925, 0.925])  

plt.show()
plt.savefig(
    os.path.join(MY_PATH, 'filename.png'),
    bbox_inches='tight',
    dpi=300,
    facecolor=fig.get_facecolor(),
    transparent=True,
)
plt.close(fig)

Answer: The solution turned out to be very simple: I had to change transparent in savefig() to False. Now, all three methods as described in my original post do as they are expected to. Case closed!
Title: Does pygbag directly interprets python in the browser or compiles it to wasm and then runs it?
Tags: python,webassembly,pygbag

Question: I was wandering through the docs of pygbag, and I couldn't find how the python scripts are actually executed from the browser.
I made a test project to look how the files created by pygbag looked like, but I couldn't really figure out what role the index.html exactly plays. It seemed to me like I couldn't find any script in it, so I supposed that it could be directly interpreted, but I'm not really sure.
There is a python script in the html file, and I found one line which seems to run the main program : await shell.runpy(main, callback=ui_callback), but I don't know whether it just executes the python script in the folder or if the script is somewhere compiled in this file.
Could anyone explain me ?

Answer: I think it would be really great if they cleaned up the project or added more docs so that the wasm build process is easier to understand and expand on.
I also looked through the index.html page and tried to do a similar thing to what you are asking about. From what I understand, there’s an Android web assembly file being built out of the python code, then it’s being run in the in the index.html file in like an iFrame or canvas (I apologize, I am not a front end expert).  I believe the pyscript stuff is for communicating to/from the python WASM file.  I do not believe that the pygame itself is being interpreted because it would likely be way way slower and too inefficient to run this way.  I have had no performance issues running a test game I made in the browser.
I personally really appreciate pygbag, and I would love to see it become even easier/better to embed pygame in the browser.  I feel like more docs & contributions would help a lot.  However, I am also a little concerned about the security of this package. I would feel a lot more comfortable contributing and using it if it had a 95+ trust score on package rating sites!
Hope this can help you!
Title: remove element by value in list Python fastest
Tags: python,list,performance,optimization

Question: what is the fastest way to remove an element from a list by its value?
I believe list.remove("element_to_be_removed") is the naive way. How can we optimize it?

Answer: A list is unordered, so finding an element by its value must be done by exhaustive search (best case O(1), worst O(n)).
Deletion also takes O(n) operations, but more precisely, it takes the number of shifts equal to the number of elements that follow the deleted one. (Plus occasional halving when the list shrinks a lot). So we have worst case O(n), best O(1), symmetrically with the search.
So if you perform the search for the key backward, the best case of removal is O(1) instead of O(n), though on average this makes absolutely no difference.
Last hint: if you have some reason to believe that the removals occur more often on a side than on the other, you'd better organize your list so that this side comes to the right (store backward if needed) and perform backward searches.
Title: remove element by value in list Python fastest
Tags: python,list,performance,optimization

Question: what is the fastest way to remove an element from a list by its value?
I believe list.remove("element_to_be_removed") is the naive way. How can we optimize it?

Answer: Finding an element in a list by value is O(n), as is removing it once you have found it.  There is no way to reduce this; it's inherent in how lists are built.
Finding and/or removing an element in a set is O(1).
Converting a list into a set is O(n).  If you have a list and you need to remove one item, converting it to a set and then removing one item doesn't get you to O(1), because the set conversion itself is an O(n) operation.  You're better off using list.remove().
If you have a list of unique items, and you anticipate needing to remove many items from it in an unordered way (say, you're going to eventually remove each item one by one), you can change that from an O(n^2) operation to an O(n) operation by converting the entire list into a set (once, in O(n)) and then removing the elements one by one after that (in O(1) per item, hence O(n) overall).
The ideal solution (if possible) is to anticipate the ways in which you'll need to use this collection of items and make it a set from the beginning if the functionality of a set is a better fit for what you're doing.
Title: Regex to extract headings and sub-headings from a pdf file using python
Tags: python,regex

Question: I have a pdffile, using pdfplumber , I have extracted all the text from it. Then I need to find all headings and sub-headings from this text. I want to use the headings and sub-headings to extract the text within those headings and sub-headings.
My headings looks  like  1. heading one 2. heading two  3. heading three  4.  heading head four is and so on - they can have maximum 5 words
My subheadings looks like  same as heading  like 1.1  heading one of one  1.2 heading two of one  2.1  heading one of two  3.2  heading two of three and so.   I am not able to do it.   I tried following but did not work  , it worked only partially ,it could find some of the heading but no sub headings
import re
# Define the pattern
pattern = r'^\s*\d+(\.\d+)?\. ((?:\b\w+\b\s*){1,5})'
# Find all matches in the text
matches = re.findall(pattern, text, re.MULTILINE)
print(matches)


and I want all the headings and sub headings to be returned in a list as mentioned above
Here is sample input data:
text= """
lotsf text text text 

1. Heading one
lots of text lots of text lots of text lot of text
123 456 text text2
0 10 text

1.1 subheading one of one

lot of text lots of text text is all
lot of text.
text and text.

1.2 subheading two of one

i m a ML enginner
i work in M
i do work in oracle also

2. Heading two

text again again text more text
holding on
backup and recovery

2.1 subheading one of two please

text text text text text

2.2 subheading two of two is

text or numbers
10  text 6345

2.3 subheading there of two

000 text 34
0 devices 
so many phone devices
""""

and expected output is :
[ 1. Heading one , 1.1 subheading one of one , 1.2 subheading two of one,2. Heading two,2.1 subheading one of two please,2.2 subheading two of two is,2.3 subheading there of two]

Answer: The reason it can't find the subheadings is (\.\d+)?\. the regex requires always having a dot after the heading number(s) and your example subheaders don't have a dot after the second number (its not 1.1. its just 1.1). To fix this edit regex to ^(\d+\.\d* (?:\w+ *){1,5})

First expand () to surround everything you want
Remove unnecessary part of regex: \s*, \b
Change digit part to \d+\.\d* to accept major/minor headings
Title: How do I install Python 2.7.10 to a conda environment?
Tags: python,package,conda,miniconda,conda-forge

Question: I need this specific version for a legacy project. This version is not included neither in the default package repository nor in the Conda Forge. How can I install this specific version to a miniconda environment? I'm using Windows 10 Enterprise

Answer: The only solution I've been able to find is to use pyenv, which is not convenient at all as whichever version you choose is global and whatever conda environment you have active is effectively ignored :/
Title: How to solve TypeError: 'type' object does not support context manager protocol?
Tags: python,speech-recognition,pywin32,pyaudio

Question: I was creating a voice assistant project but I am having problem with the line with with command in it.
The code I've written is this
import speech_recognition as sr
import win32com.client

speaker = win32com.client.Dispatch("SAPI.SpVoice")

def say(text):
    speaker.Speak(f"{text}")

def takeCommand():
    r = sr.Recognizer()
    with sr.Microphone as source:
        r.pause_threshold = 1
        audio = r.listen(source)
        query = r.recognize_google(audio, language="en-in")
        print(f"User said: {query}")
        return query

if __name__ == "__main__":
    print("VS Code")
    say("Hello I am Jarvis A.I.")
    while 1:
        print("listening...")
        text = takeCommand()
        say(text)

And the error it always gets is this
VS Code
listening...
Traceback (most recent call last):
  File "f:\Jarvis AI\main.py", line 23, in <module>
    text = takeCommand()
           ^^^^^^^^^^^^^
  File "f:\Jarvis AI\main.py", line 11, in takeCommand
    with sr.Microphone as source:
TypeError: 'type' object does not support the context manager protocol

I've installed packages in my system like pywin32, pyaudio and speechrecognition but now I don't know what to do and how to proceed.

Answer: try changing with sr.Microphone as source: to with sr.Microphone() as source:
Title: mkdocs: how to attach a downloadable file
Tags: python,markdown,mkdocs,mkdocs-material

Question: I have a mkdocs project that resembles the following:
project
├─mkdocs.yml
├─docs
│ ├─home.md
│ ├─chapter1.md
│
├─static
  ├─file.ext
  ├─image.png

I am trying to find a way to "attach" file1.ext to the build, for instance as a link in chapter1.md.
Any suggestions how to achieve that? Detail: I want the file to be downloadable on click.

Answer: In your chapter1.md file you should link your file1.ext, this file should be located in the static folder.
You can link it like:
[Link to file1](../static/file1.ext)
After this you can build your project.
Title: The empty path didn’t match any of these
Tags: python,django,django-views,django-urls

Question: I am very new in Django & trying to run my first project. But stucked in the error "The empty path didn’t match any of these." as mentioned above. In my app urls I have the following code
from django.urls import path
from . import views

urlpatterns=[
path('members/',views.members,name='members'),
]

And in my project urls I have the following code.
from django.contrib import admin
from django.urls import include, path

urlpatterns = [
path('', include('members.urls')),
path('admin/', admin.site.urls),
]

I read a number of answers of this question & found suggestion that not working for me.Seeking help to dig into the error.
Here is the full error tracbac:
Page not found (404)
Request Method: GET
Request URL:    http://127.0.0.1:8000/
Using the URLconf defined in my_tennis_club.urls, Django tried 
these URL patterns, in this order:

members/ [name='members']
admin/
The empty path didn’t match any of these.

You’re seeing this error because you have DEBUG = True in your 
Django settings file. Change that to False, and Django will display 
a standard 404 page.

Answer: You set your app path to 127.0.0.1:8000/members but you are trying to request 127.0.0.1:8000/. If you delete members/ from your path, it will work.
Title: my tkinter window ceases working when clicking child window's icon
Tags: python,tkinter

Question: import tkinter as tk

root = tk.Tk()
label = tk.Label(root, text="timer = 0")
label.pack()
nowTime=0

def nextTime():
    global root,label,nowTime
    label.config(text=nowTime)
    
    nowTime+=1
    root.after(1000, nextTime)
    
nextTime()

# child window
child = tk.Toplevel()

tk.mainloop()

i made a smoll app which generates window with label, and it increases its time
and it generates child toplevel window
when i click the "child toplevel windows icon", the main app which increases the time text ceases working, how can i fix this?
i used "grab_set()" method, but it disable all the event including "close window"
so i think it's not a fundamental solution, can i solve that without disabling all events?

Answer: When you click the icon of one of the toplevel windows you get a system menu dropped down (at least on Windows). While this menu is being shown it seems to be running it's own event loop and the the normal Tk events are not being processed. With the counter design this means a pause in your counter. You could change this to display the elapsed number of seconds if that is what you are counting so that once the menu is dismissed your display catches up.
Alternatively, in testing this it looks like as long as your dialog toplevel has a menu itself then the Tk events are processed normally and the counter will continue.