Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: How does camera distortion coefficients and camera intrinsic parameters change after image crop or resize?
Tags: python,opencv,camera,distortion,camera-intrinsics

Question: I try to make image some changes(crop, resize, undistort) and I want to know how distortion coefficients and camera intrinsinc parameters change after that.

Origin Image shape = [848, 480]
camera matrix = [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
distortion coefficients = [k1, k2, p1, p2]

crop

[848, 480] -> [582, 326]
fx, fy : no changes
cx, cy : cx -133, cy - 77
distortion coefficients -> ??

resize

[582, 326] -> [848, 480]
fx, cx -> 1.457fx, 1.457cx
fy, cy -> 1.472fy, 1.472cy
[k1, k2, p1, p2] -> [k1, k2, p1, p2] same

undistort

fx, fy, cx, cy -> same
[k1, k2, p1, p2] -> [0, 0, 0, 0]

Does anyone knows the answer?
For me I tried using my camera and calibrate some results but I don't know the exact equation.
origin

fx = 402.242923
fy = 403.471056
cx = 426.716067
cy = 229.689399
k1 = 0.068666
k2 = -0.039624
p1 = -0.000182
p2 = -0.001510

crop

fx = 408.235312 -> almost no change
fy = 409.653612 -> almost no change
cx = 297.611639 -> cx - 133
cy = 153.667098 -> cy - 77
k1 = 0.048520 -> I don't know
k2 = -0.010035 -> I don't know
p1 = 0.000943 -> I don't know
p2 = -0.000870 -> I don't know

crop_resize

fx = 598.110106 -> almost * 1.457
fy = 608.949995 -> almost * 1.472
cx = 430.389861 -> almost * 1.457
cy = 226.585804 -> almost * 1.472
k1 = 0.054762 -> I don't know
k2 = -0.025597 -> I don't know
p1 = 0.002752 -> I don't know
p2 = -0.001316 -> I don't know

undistort

fx = 404.312916 -> almost same
fy = 405.544033 -> almost same
cx = 427.986926 -> almost same
cy = 229.213162 -> almost same
k1 = -0.000838 -> almost 0
k2 = 0.001244 -> almost 0
p1 = -0.000108 -> almost 0
p2 = 0.000769 -> almost 0

Answer: All part you write as "I don't know" will be "same(not changed)".
Because Cropping and Resizing is representable with only (cx,cy,fx,fy).
Title: Hosting a flask web app with a domain I already own
Tags: python,html,css,flask,web-hosting

Question: I needed some information from you, since I haven't found much on the internet.
I recently bought a domain on godaddy.com and I wanted to use it to host my web app developed in Flask.
Given the price spent to buy the domain, I wanted to avoid spending money on hosting as well (if it could be avoided).
I tried with Netlify, but it doesn't allow you to host apps developed in Flask (it gives me everytime a 404 error), a pity because it would have been very convenient, since I could very easily configure the web app with my domain.
Can anyone tell me a possible free solution? I repeat, I already have the domain, I just need a hosting service.
Thanks in advance to those who will reply.

Answer: If you're looking for free options, your best bets are Render.com or pythonanywhere, as has already been mentioned.
Personally I'd recommend Render as I have, like you, been looking for places to host my flask apps. I used to use Heroku which was perfect but once they removed the free tier I moved away.
It depends on the complexity of your flask app as to what will work with Render, but one I'm hosting uses their PostgreSQL without any issues. It's really simple to setup too if you have your repository on GitHub with their integrations.
I have one however that doesn't properly work but I think that's more to do with limitations on the free tier and how I've coded it to work. I also tried Netlify but it doesn't seem to like Flask or Python very much.
Title: Why does adding duplicated features improve Logistic Regression accuracy?
Tags: python,scikit-learn,logistic-regression

Question: from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression

X, y = load_iris(return_X_y=True)

for i in range(5):
    X_redundant = np.c_[X,X[:,:i]] # repeating redundant features
    print(X_redundant.shape)
    clf = LogisticRegression(random_state=0,max_iter=1000).fit(X_redundant, y)
    print(clf.score(X_redundant, y))

Output
(150, 4)
0.9733333333333334
(150, 5)
0.98
(150, 6)
0.98
(150, 7)
0.9866666666666667
(150, 8)
0.9866666666666667

Question: Why is the score (default being Accuracy) increasing as more redundant features are added for Logistic Regression?
I expect the score to remain the same, by drawing analogies from LinearRegression's behaviour.
If it was LinearRegression, the score (default R2) will not change because as more columns are added because LinearRegression will evenly distribute coef between each of the 2 redundant coefficients
from sklearn.datasets import load_iris
from sklearn.linear_model import LinearRegression

X, y = load_iris(return_X_y=True)
X, y = X[:,:-1],X[:,-1]
 
for i in range(4):
    X_redundant = np.c_[X,X[:,:i]] # repeating redundant features
    print(X_redundant.shape)
    clf = LinearRegression().fit(X_redundant, y)
    print(clf.score(X_redundant, y))
    print(clf.coef_)

Output
(150, 3)
0.9378502736046809
[-0.20726607  0.22282854  0.52408311]
(150, 4)
0.9378502736046809
[-0.10363304  0.22282854  0.52408311 -0.10363304]
(150, 5)
0.9378502736046809
[-0.10363304  0.11141427  0.52408311 -0.10363304  0.11141427]
(150, 6)
0.9378502736046809
[-0.10363304  0.11141427  0.26204156 -0.10363304  0.11141427  0.26204156]

Answer: This is because LogisticRegression applies regularization by default. Set penalty="none" or penalty=None (depending on your version of sklearn) and you should see the behavior you expected.
Title: Dropping duplicate rows in a Pandas DataFrame based on multiple column values
Tags: python,pandas,drop-duplicates

Question: In a dataframe I need to drop/filter out duplicate rows based the combined columns A and B. In the example DataFrame
    A   B   C   D
0   1   1   3   9
1   1   2   4   8
2   1   3   5   7
3   1   3   4   6
4   1   4   5   5
5   1   4   6   4

rows 2 and 3, and 5 and 6 are duplicates and one of them should be dropped, keeping the row with the lowest value of
2 * C + 3 * D

To do this, I created a new temporary score column, S
df['S'] = 2 * df['C'] + 3 * df['D']

and finally to return the index of the minimum value for S
df.loc[df.groupby(['A', 'B'])['S'].idxmin()]
del ['S']

The result is
    A   B   C   D
0   1   1   3   9
1   1   2   4   8
3   1   3   4   6
5   1   4   6   4

However, is there a more efficient way of doing this, without having to add (and later drop) a new column?

Answer: df.drop_duplicates(subset=['A','B'],inplace=True)
by default, drop_duplicates drops rows (axis=0).
Title: How do I remedy this attributeerror in terminal while trying to execute a gradio interface?
Tags: python,macos,terminal,gradio,gr

Question: in Mac terminal, gotten similar errors in Jupyter. When I try and run this python app. I keep getting these errors.

line 3, in 
import gradio as gr line 6, in 
demo = gr.Interface(fn=greet, inputs="text", outputs="text") AttributeError: partially initialized module 'gradio' has no attribute
'Interface' (most likely due to a circular import)

I haven't used gradio much so any advice would be much appreciated.
from gpt_index import (
    SimpleDirectoryReader,
    GPTListIndex,
    GPTSimpleVectorIndex,
    LLMPredictor,
    PromptHelper,
)
from langchain import OpenAI
import gradio as gr
import sys
import os

os.environ["OPENAI_API_KEY"] = "My API Key"


def construct_index(directory_path):
    max_input_size = 4096
    num_outputs = 512
    max_chunk_overlap = 20
    chunk_size_limit = 600

    prompt_helper = PromptHelper(
        max_input_size,
        num_outputs,
        max_chunk_overlap,
        chunk_size_limit=chunk_size_limit,
    )

    llm_predictor = LLMPredictor(
        llm=OpenAI(
            temperature=0.7, model_name="text-davinci-003", max_tokens=num_outputs
        )
    )

    documents = SimpleDirectoryReader(directory_path).load_data()

    index = GPTSimpleVectorIndex(
        documents, llm_predictor=llm_predictor, prompt_helper=prompt_helper
    )

    index.save_to_disk("index.json")

    return index


def chatbot(input_text):
    index = GPTSimpleVectorIndex.load_from_disk("index.json")
    response = index.query(input_text, response_mode="compact")
    return response.response


iface = gr.Interface(
    fn=chatbot,
    inputs=gr.inputs.Textbox(lines=7, label="Enter your text"),
    outputs="text",
    title="Chat",
)

index = construct_index("database")
iface.launch(share=True)

I tried the code above and was expecting it to launch the gradio interface.

Answer: Check your Python file name, if it's gradio.py rename it to something else
Title: Getting this error when tried to install using pip install anymail
Tags: python,django,django-anymail

Question: ERROR: Could not find a version that satisfies the requirement anymail (from versions: none)
ERROR: No matching distribution found for anymail
Can anyone help me with this

Answer: try upgrading your pip pip install --upgrade pip
note: you may need to check your other packages after the upgrade
Title: How should I using torch.compile properly?
Tags: python,machine-learning,optimization,deep-learning,pytorch

Question: I'm currently trying to use pytorch 2.0 for boosting training performance with my project. And I've heard that torch.compile might be boost some models.
So my question (for now) is simple; how should I use torch.compile with large model?
Such as, should I use torch.model like this?
class BigModel(nn.Module):
    def __init__(self, ...):
        super(BigModel, self).__init__()
        self.model = nn.Sequential(
            SmallBlock(), 
            SmallBlock(), 
            SmallBlock(), 
            ...
        )
        ...

class SmallBlock(nn.Module):
    def __init__(self, ...):
        super(SmallBlock, self).__init__()
        self.model = nn.Sequential(
            ...some small model...
        )

model = BigModel()
model_opt = torch.compile(model)

,or like this?
class BigModel(nn.Module):
    def __init__(self, ...):
        super(BigModel, self).__init__()
        self.model = nn.Sequential(
            SmallBlock(), 
            SmallBlock(), 
            SmallBlock(), 
            ...
        )
        ...

class SmallBlock(nn.Module):
    def __init__(self, ...):
        super(SmallBlock, self).__init__()
        self.model = nn.Sequential(
            ...some small model...
        )
        self.model = torch.compile(self.model)

model = BigModel()
model_opt = torch.compile(model)

For summary,

Should compile each layer? or torch.compile do this automatically?
Is there any tips for using torch.compile properly?

Thanks
To be honest, I tried both, but there are no differences..
And also, it didn't speed up dramatically, I just checked speed up rate for my model just about 5 ~ 10%.

Answer: the default mode of torch.compile does not seem to work, but it has another mode that can really accelerate your model.
"""
torch.compile(${yourmodel}, mode="reduce-overhead")
"""
Title: Why does using df[[column_name]] dislays a DataFrame?
Tags: python,pandas,dataframe,numpy,data-analysis

Question: I was just asking myself: I understand that calling df[column_name] displays a Series because a DataFrame is built of different arrays.
Though, why does calling df[[column_name]] (column_name being only one column) returns a DataFrame and not a Series ? I'm ont sure to understand the logic behing how Pandas was built here
Thanks :)
I was trying to explain to my students why calling a list of one element displays a dataframe and not a Series, but did not manage

Answer: It may happend because when you give single column_name as a string it takes it perform selection and return single value based on the search key column_name.
But when you provide same column_name contained in a list it tries to fetch all the keys of the list which in one in this case. Hence resulting a dataframe.
I guess they are using some standard logic to return dataframe if list is provided irrespective of length of list.
import pandas as pd
df = pd.DataFrame(columns=["a","b","c"],
data=[[1,4,7],[2,5,8],[3,6,9]])
column_name = "a"
print(type(df[column_name]))
print(type(df[[column_name]]))
output:
<class 'pandas.core.series.Series'>
<class 'pandas.core.frame.DataFrame'>
Title: I can't run python script in visual studio code
Tags: python

Question: I have problem in running python script in visual studio code, in other IDE I dont have issue I run the same python script without any issue.
in the terminal I got this message
conda activate base
conda : The term 'conda' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of
the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1

conda activate base

  + CategoryInfo          : ObjectNotFound: (conda:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException

Answer: Sounds like there is an issue with the interpreter.

Open the Command Palette in Visual Studio Code by pressing Ctrl + Shift + P
Type "Python: Select Interpreter" and select it.
Choose the conda environment that you want to use from the list of available interpreters.
After selecting the interpreter, close and reopen VSC (for changes to take effect).

Once you have the correct interpreter, you should be able to run the script in the terminal.
Title: Python discord bot not joining voice channel, but without sending an error
Tags: python,discord

Question: I made a music bot for a discord server one month ago, using discord.py. It worked well until one week ago, but now when I use the join command the bot fails to connect to the voice channel. Here is the code of the command :
class MonClient(discord.Client):
      
   async def on_ready(self):
      self.voice = None # contains the <VoiceClient> object created by <chan.connect> in the join command

   async def on_message(self,message):
      if message.author.bot:
         pass
      elif message.content[0] == "!": # the command prefix is "!"
         parts = message.content.split(" ")
         command = parts[0][1:]
         
         if command == "join":
            if message.author.voice != None and self.voice == None:
               chan = message.author.voice.channel
               await message.channel.send("before connecting to voice channel")
               self.voice = await chan.connect()
               await message.channel.send("after connecting to voice channel")
            else:
               await message.channel.send("Bot already connected to voice channel or user not connected to voice channel")

More precisely, the bot successfully sends "before connecting to voice channel" in the text channel, then it is shown to have joined to the voice channel by the Discord UI (its image is shown in the list of members connected to the voice channel, there is the sound of connection to the voice channel). But the second message "after connecting to voice channel" is never sent. We can still use the bot's other commands, but the variable self.voice still contains None, so obviously the bot can't play music.
I suppose the asynchronous connect method stops on something indefinitely, but I can't understand what, because there is no error message, not even the TimeoutError described by the documentation. The internet connection seems fine, and as I said the bot worked well for two or three weeks, so now I really don't know how to better understand what blocks the connection.
There are some other questions on Stackoverflow for similar issues, but the proposed answers had no relation with my problem : an answer proposed another method to get the voice channel object to connect to, but I already get the good object since the bot is at least shown to connect to the voice channel ; and another answer had to do with the @client.command() lines, which I don't use.

Answer: Make sure your library is up to date, discord made a change recently. It worked for me.
pip install -U discord
Title: how to decode Redis ft search query?
Tags: python,redis,redis-py

Question: So I am new to redis and I am trying to learn everything I possible can and later want to add this for caching in my website.
But, I am currently encountering a very small problem, I have no idea of any function how to decode the return query from the ft function.
The code is below:
import redis
import json
from redis.commands.json.path import Path
from redis.commands.search.query import Query,NumericFilter
r=redis.Redis(host="192.168.196.75",port=6379)
user1 = {
        "name": "Paul John",
        "email": "paul.john@example.com",
        "age": 42,
        "city": "London",
        "id":1,
        "username":"Paul Walker",
}

# r.json().set("user:1",Path.root_path(),user1)

loc=r.ft("Idx").search(query="@username:Paul")
print(loc)

The output I am getting is Result{1 total, docs: [Document {'id': 'user:1', 'payload': None, 'json': '{"name":"Paul John","email":"paul.john@example.com","age":42,"city":"London","id":1,"username":"Paul Walker"}'}]}
As you can see everything is working fine, But I am not able to derive the dictioniary that has the information, I can try to make many steps like splitting and slicing but I think there is a function for this which I am not able to find.
It would be very helpful If anyone knows what that is. Thank you for reading my question

Answer: for retrieving just the documents you can use the .docs such as:
loc=r.ft("Idx").search(query="@username:Paul").docs retrieving all results or .docs[0] for the first element and on
Title: What is the difference between exhausting a generator using "for i in generator" and next(generator)
Tags: python,generator

Question: I want to learn how to use the return value of a generator (but this not what I'm concerned with now).
After searching, they said that I can get the return value from StopIteration when the generator is exhausted, so I tested it with the following code:
def my_generator():
    yield 1
    yield 2
    yield 3
    return "done"


def exhaust_generator(_gen):
    print("===============================================\n")
    print("exhaust_generator")
    try:
        while True:
            print(next(_gen))
    except StopIteration as e:
        print(f"Return value: '{e.value}'")


def exhaust_generator_iter(_gen):
    print("===============================================\n")
    print("exhaust_generator_iter")
    try:
        for i in _gen:
            print(i)
        print(next(_gen))
    except StopIteration as e:
        print(f"Return value: {e.value}")


gen = my_generator()
gen2 = my_generator()
exhaust_generator(gen)
exhaust_generator_iter(gen2)


===============================================

exhaust_generator
1
2
3
Return value: 'done'
===============================================

exhaust_generator_iter
1
2
3
Return value: None


As you can see, the return value is different between the two versions of exhausting the generator and I wonder why.
Searched google but it has not been helpful.

Answer: In your first example with the while, you're catching the first StopIteration exception raised by the generator when it's initially exhausted.
In your second example with the for, you're catching a subsequent StopIteration exception raised by calling next() a second time after the generator has already been exhausted (by the for, which caught that initial exception).
Title: Facebook API is sending a GET request instead of a POST to my Chatbot
Tags: python,api,facebook,flask,request

Question: When using Facebook Messenger to send a chat message to my Chatbot, I don't get a reply. Checking the server logs show that I am receiving a GET request whereas it should be a POST request.
This is weird because the chatbot works when tested by running the file locally, whereas upon deploying it on Digital Ocean Droplet, it receives it as a GET request.
The deployed version works if i explicitly send a POST request via Postman (while simulating a facebook messaging request body)
Is there any reason Facebook is sending / I am receiving it as a GET request instead of a POST?
I tried to test the file locally and it works, and through simulating requests with Postman. It just doesn't work if the request is sent via a message on Facebook Messenger

Answer: Made a mistake before, it was indeed related to the redirections as stated by CBroe.
POST Endpoint was made for url with a trailing slash whereas the callback set on facebook developers was set to handle no trailing slash. Just had to fix that and redirection issue was resolved!
Title: AttributeError: module 'networkx' has no attribute 'info'
Tags: python-3.x,networkx,nomachine-nx

Question: import networkx as nx
G = nx.read_edgelist('webpages.txt', create_using=nx.DiGraph())
print(nx.info(G))

AttributeError: module 'networkx' has no attribute 'info'

print(nx.info(G)) without the error

Answer: You can also just print(G) to get the graph properties
Title: OpenAI ChatGPT (GPT-3.5) API error 429: "You exceeded your current quota, please check your plan and billing details"
Tags: python,prompt,openai-api,completion,chatgpt-api

Question: I'm making a Python script to use OpenAI via its API. However, I'm getting this error:
openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details
My script is the following:
#!/usr/bin/env python3.8
# -*- coding: utf-8 -*-

import openai
openai.api_key = "<My PAI Key>"

completion = openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "user", "content": "Tell the world about the ChatGPT API in the style of a pirate."}
  ]
)

print(completion.choices[0].message.content)

I'm declaring the shebang python3.8 because I'm using pyenv. I think it should work, since I did 0 API requests, so I'm assuming there's an error in my code.

Answer: I was facing the same error, and for me the steps were:

Add a credit or debit card in payment methods.
Generate a new API key in user preferences.
(Optional) Delete the old API key.
Be sure to set limits to not incur in charges

This are the limits for gpt-3.5-turbo RPM 3,500 TPM 90,000
Hope it helps.
Title: OpenAI ChatGPT (GPT-3.5) API error 429: "You exceeded your current quota, please check your plan and billing details"
Tags: python,prompt,openai-api,completion,chatgpt-api

Question: I'm making a Python script to use OpenAI via its API. However, I'm getting this error:
openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details
My script is the following:
#!/usr/bin/env python3.8
# -*- coding: utf-8 -*-

import openai
openai.api_key = "<My PAI Key>"

completion = openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "user", "content": "Tell the world about the ChatGPT API in the style of a pirate."}
  ]
)

print(completion.choices[0].message.content)

I'm declaring the shebang python3.8 because I'm using pyenv. I think it should work, since I did 0 API requests, so I'm assuming there's an error in my code.

Answer: Just create a new API Key and use it, it worked for me
Title: OpenAI ChatGPT (GPT-3.5) API error 429: "You exceeded your current quota, please check your plan and billing details"
Tags: python,prompt,openai-api,completion,chatgpt-api

Question: I'm making a Python script to use OpenAI via its API. However, I'm getting this error:
openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details
My script is the following:
#!/usr/bin/env python3.8
# -*- coding: utf-8 -*-

import openai
openai.api_key = "<My PAI Key>"

completion = openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "user", "content": "Tell the world about the ChatGPT API in the style of a pirate."}
  ]
)

print(completion.choices[0].message.content)

I'm declaring the shebang python3.8 because I'm using pyenv. I think it should work, since I did 0 API requests, so I'm assuming there's an error in my code.

Answer: I encountered a similar issue and found a solution that worked for me. I first canceled my paid account and renewed it with a different payment method. Next, I went to the 'API Keys' section, selected my organization under the 'Default Organizations' dropdown, and saved the changes. This action reset my soft limit, but I still needed to create a new API key to resolve the issue completely.

Cancel paid account and recreate with new payment method
Confirm Organization
Create new API Key
Title: Django datetime field prints out unsupported operand type error
Tags: python,django,datetime

Question: I have a @property in my django model which prints out True or False based on datetime difference. Even though it works when I am unittesting it I get error
TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'datetime.date'

class MyModel(models.Model):
    ...
    created_at = models.DateTimeField(auto_now_add=True)
    
    @property
    def is_expired(self):
        if datetime.now(tz=utc) - self.created_at > timedelta(hours=48):
            return True
        return False


How do I fix this? Every answer I checked does not help me .

Answer: The traceback error is explaining the problem to you:

unsupported operand type(s) for -: 'datetime.datetime' and 'datetime.date'

This says that you cannot subtract a datetime.datetime from a datetime.date. In your case, self.created_at appears to be a date but your use of datetime.now() and timedelta() suggest that you really want to be dealing with datetimes.
Therefore, the solution is to either

make sure that created_at is a datetime, or
change datetime.now() to date.today() and change timedelta(hours=48) to timedelta(days=2)

These will behave a little differently so pick one depending on whether things expire after two calendar days or in 48 hours.
Title: ValueError: Shapes (None, 20, 9) and (None, 9) are incompatible
Tags: python,tensorflow,keras,artificial-intelligence

Question: This is my code
import tensorflow as tf
from tensorflow import keras
from keras import layers
# definition  question and answer
questions = [
    "你好嗎？",
    "今天天氣如何？",
    "你有什麼興趣？",
    "你喜歡什麼食物？"
]

answers = [
    "我很好，謝謝你。",
    "今天天氣非常好。",
    "我喜歡看書和打電子遊戲。",
    "我喜歡吃中式食物，特別是炒飯。"
]
# 將問題和回答轉換成數字
tokenizer = keras.preprocessing.text.Tokenizer()
tokenizer.fit_on_texts(questions + answers)


question_seqs = tokenizer.texts_to_sequences(questions)
answer_seqs = tokenizer.texts_to_sequences(answers)

# 將問題和回答填充到相同的長度
max_len = 20

question_seqs_padded = keras.preprocessing.sequence.pad_sequences(question_seqs, maxlen=max_len)
answer_seqs_padded = keras.preprocessing.sequence.pad_sequences(answer_seqs, maxlen=max_len)

# 定義模型
model = keras.Sequential()
model.add(layers.Embedding(len(tokenizer.word_index) + 1, 50, input_length=max_len))
model.add(layers.LSTM(64))
model.add(layers.Dense(len(tokenizer.word_index) + 1, activation='softmax'))
# 編譯模型
model.compile(loss='categorical_crossentropy', optimizer='adam')


# 訓練模型
model.fit(question_seqs_padded, keras.utils.to_categorical(answer_seqs_padded, num_classes=len(tokenizer.word_index)+1), epochs=100, batch_size=32)

and this is it ran out of error
    ValueError: Shapes (None, 20, 9) and (None, 9) are incompatible

I tryed to fix the Shapes (None, 20, 9) and (None, 9) are incompatible
model.fit(question_seqs_padded, keras.utils.to_categorical(answer_seqs_padded, num_classes=len(tokenizer.word_index)+1), epochs=100, batch_size=32)

I try to delete the answer_seqs_padded to incompatible (None, 9),but it is still not work.

Answer: In this case, you need to return the whole sequence in lstm, so just use:
layers.LSTM(64, return_sequences=True) instead. If you don't use return_sequences=True, it will just return the last output.
Title: TDD modifying my test to make my code pass
Tags: python,flask,tdd

Question: I'm learning Test Driven Development, i'm struggling a little bit, seems like my brain wants to build solution algorithm, and not the needed test to build my algorithm. I can barely formalize unit test, and i go back and forth to change my tests.
I come to a point where i make some adjustement in my code, and then i'm trying to modify my tests to make them pass! It should be the other way around.
i had initially this piece of code for test
    def test_is_zero_in_range(self):
        self.assertRaises(Exception, self._provServerPrepare.is_in_allow_range, 0)

so i write it up the code needed
    def is_in_allow_range(self, value: str) -> int:
        value = int(value)
        if value in MULTIPLY_PROV_RANGE:
            return value
        raise Exception('Value out of range')

as i'm working with flask, i changed my code to get a return with a message and an error code, rather than a raise exception
    def is_in_allow_range(self, value: str) -> int:
        value = int(value)
        if value in MULTIPLY_PROV_RANGE:
            return value
        return {"message": f"Value out of range"}, 400 

In this situation i have to change my tests to make it work, i'm not sure, but i feel like i'm messing with something, this doesn't seem right to me to do it that way.
Does anyone can help me on this, or have any resources for me to read/watch?
Thx

Answer: You're discovering your actual requirements and evolving your way there, which is one of the primary goals and benefits of TDD. There is nothing wrong with changing your mind.
If you have a dozen tests making a call, and then you need to change how that call works, you have two options:

Use an IDE like PyCharm that supports the Change Signature refactoring.
Extract a helper method on the test side that makes the call. Then you only have to change one place. Such helpers often move from test code to production code.

Another thing to be aware of with TDD: Don't start with an edge case. Instead, capture a stripped-down subset of what you want. At Industrial Logic, we call this Essence First. Throwing an exception is likely not the essence of the behavior you want to TDD, so don't start there. Instead, start with a minimal expression of the "happy path."
Title: TDD modifying my test to make my code pass
Tags: python,flask,tdd

Question: I'm learning Test Driven Development, i'm struggling a little bit, seems like my brain wants to build solution algorithm, and not the needed test to build my algorithm. I can barely formalize unit test, and i go back and forth to change my tests.
I come to a point where i make some adjustement in my code, and then i'm trying to modify my tests to make them pass! It should be the other way around.
i had initially this piece of code for test
    def test_is_zero_in_range(self):
        self.assertRaises(Exception, self._provServerPrepare.is_in_allow_range, 0)

so i write it up the code needed
    def is_in_allow_range(self, value: str) -> int:
        value = int(value)
        if value in MULTIPLY_PROV_RANGE:
            return value
        raise Exception('Value out of range')

as i'm working with flask, i changed my code to get a return with a message and an error code, rather than a raise exception
    def is_in_allow_range(self, value: str) -> int:
        value = int(value)
        if value in MULTIPLY_PROV_RANGE:
            return value
        return {"message": f"Value out of range"}, 400 

In this situation i have to change my tests to make it work, i'm not sure, but i feel like i'm messing with something, this doesn't seem right to me to do it that way.
Does anyone can help me on this, or have any resources for me to read/watch?
Thx

Answer: It happens all the time that after you write a test, you find that you need to change the return value from a function (or the exception that it raises). That's normal. The TDD way to address this is to change the test first to reflect the new correct return value. See that test fail, and then update your code to make it pass.
So, in your example, when you switched to flask, or when you realized that you needed to return a message and error code rather than raise an exception, you would first modify test_is_zero_in_range to require the message and error code, see it fail, and then update is_in_allow_range to make it pass. "Test first."
By the way, here are some alternative names for your test and your function, which you might like better:
test_zero_is_out_of_range  or  test_zero_should_be_out_of_range
is_in_allowed_range
This test name is more self-explanatory, since it states the intent of the test.
Title: Polars Modify Many Columns Based On Value In Another Column
Tags: python,python-polars

Question: Say I have a DataFrame that looks like this:
df = pl.DataFrame({
  "id": [1, 2, 3, 4, 5],
  "feature_a": np.random.randint(0, 3, 5),
  "feature_b": np.random.randint(0, 3, 5),
  "label": [1, 0, 0, 1, 1],
})

┌─────┬───────────┬───────────┬───────┐
│ id  ┆ feature_a ┆ feature_b ┆ label │
│ --- ┆ ---       ┆ ---       ┆ ---   │
│ i64 ┆ i64       ┆ i64       ┆ i64   │
╞═════╪═══════════╪═══════════╪═══════╡
│ 1   ┆ 2         ┆ 0         ┆ 1     │
│ 2   ┆ 1         ┆ 1         ┆ 0     │
│ 3   ┆ 2         ┆ 2         ┆ 0     │
│ 4   ┆ 1         ┆ 0         ┆ 1     │
│ 5   ┆ 0         ┆ 0         ┆ 1     │
└─────┴───────────┴───────────┴───────┘

I want to modify all the features columns based on the value in the label column, producing a new DataFrame.
┌─────┬───────────┬───────────┐
│ id  ┆ feature_a ┆ feature_b │
│ --- ┆ ---       ┆ ---       │
│ i64 ┆ i64       ┆ i64       │
╞═════╪═══════════╪═══════════╡
│ 1   ┆ 1         ┆ 1         │
│ 2   ┆ 0         ┆ 0         │
│ 3   ┆ 0         ┆ 0         │
│ 4   ┆ 1         ┆ 1         │
│ 5   ┆ 1         ┆ 1         │
└─────┴───────────┴───────────┘

I know I can select all the features columns by using regex in the column selector
pl.col(r"^feature_.*$")

And I can use a when/then expression to evaluate the label column
pl.when(pl.col("label") == 1).then(1).otherwise(0)

But I can't seem to put the 2 together to modify all the selected columns in one fell swoop. It seems so simple, what am I missing?

Answer: Here's one way:
Recently support was added for more ergonomic arguments in a lot of methods, including with_columns and select. Since they now can take any number of keyword arguments acting like an alias at the end (e.g. setting the new column name), we can construct a dict of the columns to overwrite and pass it in (with unpacking) like so:
df.select('id', **{col : 'label' for col in df.columns if col.startswith('feature')})
In this simple case no when/then is needed for the label column, but in general any expression evaluating to a column of the same height as id can go into this dict comprehension.
Title: Can someone explain this "for" loop in python?
Tags: python,windows,list,spyder

Question: I don't understand the meaning of this for loop, specifically the range arguments:
for element in range(len(text1)-1,-1,-1):
    print(text1[element])

Answer: Answer above says it all. But I can add that making the second arguments of range -2 or -3 and so on, will make your loop goes multiple times. So once it's done printing all the elements of your list, it will redo the same operation as many times you specified.
Title: Can someone explain this "for" loop in python?
Tags: python,windows,list,spyder

Question: I don't understand the meaning of this for loop, specifically the range arguments:
for element in range(len(text1)-1,-1,-1):
    print(text1[element])

Answer: The loop will print the element of the list text1 at position element, but the for loop will make it print starting first from the last element of your list text1.
The syntax of the range is the following: range(start,stop,step). The reason why the loop starts at len(text1)-1 is because list in python are indexed not starting from 1 but from 0. So if your list has n elements, it will be indexed from 0 to n-1.
I suggest you try to change the values used in range to understand what is happening.
Title: FastAPI responds with required field is missing
Tags: python,fastapi,starlette

Question: I'm following a simple tutorial from the FastAPI docs. I was already using SQLAlchemy in this project just added the fastapi dependency and trying to run it, here's my code:
import re
import json
import copy
import traceback
import urllib.parse
from models import *
from mangum import Mangum
from datetime import datetime
from sqlalchemy.orm import Session
from sqlalchemy import *
from sqlalchemy import create_engine
from collections import defaultdict
from fastapi import FastAPI, Depends, Request, Response

app = FastAPI()


@app.middleware("http")
async def db_session_middleware(request, call_next):
    response = Response("Internal server error", status_code=500)
    try:
        engine = create_engine(
            "some db"
        )
        base.metadata.create_all(engine)
        request.state.db = Session(engine)
        response = await call_next(request)
    finally:
        request.state.db.close()
    return response


def get_db(request):
    return request.state.db


@app.get("/")
def get_root():
    return {"Status": "OK"}


@app.get("/products/{sku}")
def get_product(sku, db=Depends(get_db)):
    pass


@app.get("/products", status_code=200)
def get_products(page: int = 1, page_size: int = 50, db: Session = Depends(get_db)):
    try:
        result, sku_list = [], []
        for row in (
            db.query(Product, Image)
            .filter(Product.sku == Image.sku)
            .limit(page_size)
            .offset(page * page_size)
        ):
            if row[0].sku not in sku_list:
                result.append(
                    {
                        "sku": row[0].sku,
                        "brand": row[0].brand,
                        "image": row[1].image_url,
                        "title": row[0].product_title,
                        "price": row[0].original_price,
                        "reviewCount": row[0].total_reviews,
                        "rating": row[0].overall_rating,
                    }
                )
                sku_list.append(row[0].sku)

        print(f"Result: {result}")
        return {"body": {"message": "Success", "result": result}, "statusCode": 200}
    except Exception as err:
        print(traceback.format_exc(err))
        return {
            "body": {"message": "Failure", "result": traceback.format_exc(err)},
            "statusCode": 500,
        }

When I hit the products endpoint using this url:
http://127.0.0.1:8000/products?page=1&page_size=100

I'm getting this response:
{"detail":[{"loc":["query","request"],"msg":"field required","type":"value_error.missing"}]}

I'm not sure what it means by query and request missing in the response. What am I doing wrong?

Answer: declare request as argument.
not  async def endp(request, ...), but async def endp(request: Request, ...)
import Request from fastapi
from fastapi import Request
Title: What's the time complexity of this code with a branch?
Tags: python,math,time-complexity,big-o

Question: def p(a, b):
    if b == 0:
        return 1
    if b % 2 == 0:
        return p(a, b//2) * p(a, b//2)
    return a * p(a, b-1)


I know that it's O(n) if the code is:
def p(a, b):
    if b == 0:
        return 1
    return p(a, b//2) * p(a, b//2)

However, when there is a return a * p(a, b-1) at the end, I'm not sure whether the time complexity will change.

Answer: Ok I did some maths, and you can prove (by a recurrence) that, when you call p(a,b), the total number of calls to p() is <= 3b - 1 for b >= 1, which indeed makes the time complexity O(b).
I won't give the full details, but the gist of it is that (calling N(b) the total number of calls to p()) : N(1) = 2; N(b+1) =  1 + N(b) if b is even and 1 + 2 * N(b/2) if b is odd. You can verify this, then do the recurrence.
Interestingly, the worst case scenario, where N(b) = 3b - 1, happens when b is a power of 2.
Title: Jupyterlab extension gives a function not found error
Tags: python,jupyter-notebook,jupyter-lab

Question: I have issues with jupyter extensions on ArchLinux. In particular, I get the following error:
[W 2023-04-01 18:34:36.504 ServerApp] A `_jupyter_server_extension_points` function was not found in jupyter_nbextensions_configurator. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[W 2023-04-01 18:34:36.493 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.

How can I get rid of this error/warning? I tried removing the functions with Pip, but it did not work.
Any ideas?

Answer: This is not an error, but a warning aimed at developers of notebook extensions: jupyter_nbextensions_configurator and notebook_shim respectively, not at a user like you. You do not need to do anything.
It's worth pointing out that the next version of Jupyter Notebook (v7) will not require jupyter_nbextensions_configurator anymore as it will come with new extension manager (and a completely new extension ecosystem shared with JupyterLab), so there is very little incentive to address this deprecation warning in jupyter_nbextensions_configurator as it will cease to be relevant soon.
Title: Taking the mean of a row of a pandas dataframe with NaN and arrays
Tags: python,arrays,pandas,dataframe,numpy

Question: Here is my reproducible example:
import pandas as pd
import numpy as np
df = pd.DataFrame({'x' : [np.NaN, np.array([0,2])], 'y' : [np.array([3,2]),np.NaN], 'z' : [np.array([4,5]),np.NaN], 't' : [np.array([3,4]),np.array([4,5])]})

I would like to compute the mean array for each row excluding NaN
I have tried df.mean(axis=1) which gives NaN for both row. This is particularly surprising to me as df.sum(axis=1) appears to be working as I would have expected.
[df.loc[i,:].mean() for i in df.index] does work but I am sure there is a more straightforward solution.

Answer: Your DataFrame uses the object dtype which is always a bit of a bodge.  It's slower than native types, and doesn't always behave the way you'd expect.
Since Pandas removed the "Panel" type which was used for 3D data, I'd recommend you not store this data in a DataFrame.  Instead, store it in a 3D NumPy array, then you can use np.nanmean() to easily calculate averages while ignoring NaN.
Title: My HTML is not sending the POST method to the flask app function
Tags: python,html,sqlite,flask

Question: What's happening:



I have an HTML code that is sending a POST request to the flask code. It's for a Login page. (I'm using SQLite3 and Flask.)


The HTML code:


   <div class="col">
            <div class="login-box">
            <h2>Register</h2>
            <form method="post" action="{{ url_for('register_post') }}">
                <div class="user-box">
                    <input type="email" name="email" placeholder="Email Address" required>
                </div>
                <div class="user-box">
                    <input type="password" name="password" placeholder="Password" required>
                </div>
                <div class="user-box">
                    <input type="text" name="username" placeholder="Username" required>
                </div>
                <div class="button-form">        
                    <a id="submit" href="{{url_for('register_post')}}">Submit</a>
                    <div id="register">
                        Already Have An Account ?
                        <a href="{{url_for('login')}}">Login Now !</a>
                    </div>
                </div>
            </form>
        </div>
    </div>


The Python code:


@app.route('/register', methods=['POST'])
def register_post():
    print("Got a data")
    username = request.form['username']
    email = request.form['email']
    password = request.form['password']
    c.execute('''INSERT INTO users (email, username, password) VALUES (?,?,?)''', (email, username, password))
    conn.commit()
    conn.close()

    return redirect(url_for('login'))

There isn't a traceback which (maybe) means the function hasn't even been called.

Answer: I have not worked with Flask, but have with Django.
I see you got an <a> tag with href="{{url_for('register_post')}}", why are you trying to redirect to 'https://example.com/register'? By redirecting you are making GET request whereas your register_post function only handles POST method.

You described <form> with method="post" and action="{{url_for('register_post')}}", this means that the function will work whenever someone submits the form, but for now it's possible when you click the 'enter' key when focused on any input of the form. Another way to do it is to add <button type="submit">yout text</button> or <input type="submit" value="your text" /> instead of <a id="submit"> tag.

There is only two methods of HTTP Request in HTML those being POST and GET.

GET request occurs when you are navigating via URL or submit a form with method="GET" which is default and you do not have to specify it.
POST method occurs when someone SUBMITS a FORM with POST method described.


If you want to access any other HTTP Requests kind of PUT, DELETE, HEAD, OPTIONS, PATCH, then you need JavaScript for it.
Title: ImportError: cannot import name 'CLOSED' from 'websockets.connection'
Tags: python,websocket,rasa

Question: I installed rasa on a virtual environment on windows. But while I am trying to check either rasa is installed or not, it is showing an error that says-
ImportError: cannot import name 'CLOSED' from 'websockets.connection'
I have reinstalled rasa, and installed websockets. But still getting the error.
Python version is 3.10.2
Can anyone help me to solve this problem?

Answer: Try installing websockets version 10.0 like this:
pip install websockets==10.0
It should help (it helped me)
Title: ValueError: in user code: ValueError: No gradients provided for any variable for Autoencoder Tensorflow
Tags: python,tensorflow,autoencoder

Question: I have the above code and I have the below error during my fit
ValueError: No gradients provided for any variable: (['dense_12/kernel:0', 'dense_12/bias:0', 'dense_13/kernel:0', 'dense_13/bias:0'],). Provided grads_and_vars is ((None, <tf.Variable 'dense_12/kernel:0' shape=(784, 2) dtype=float32>), (None, <tf.Variable 'dense_12/bias:0' shape=(2,) dtype=float32>), (None, <tf.Variable 'dense_13/kernel:0' shape=(2, 784) dtype=float32>), (None, <tf.Variable 'dense_13/bias:0' shape=(784,) dtype=float32>)).
I tried to change encoder and decoder and It didn't solve the problem. I have an autoencoder problem that is why i used X_train and X_test dataset instead of y_train and y_test dataset.
How can I solve this error?
Thank you for your feedbacks!

class autoencoder(Model):
  def __init__(self, output_dim = 2): 
    super(autoencoder, self).__init__()
    self.output_dim = output_dim   
    self.encoder = tf.keras.Sequential([
      layers.Flatten(),
      layers.Dense(2, activation='relu')
    ])
    self.decoder = tf.keras.Sequential([
      layers.Dense(784, activation='relu')
     
    ])
  def encoder_call(self, inputs):
    inputs = Flatten()(inputs)
    return self.encoder(inputs) 
  def call(self, x, train = None):
    x = Flatten()(x)
    encoded = self.encoder(x)
    decoded = self.decoder(encoded)
    return tf.reshape(x,(-1,28,28)) 


model = autoencoder()
model.compile(optimizer='adam', loss=losses.MeanSquaredError())

model.fit(X_train, X_train,
                epochs=1,
                shuffle=True,
                validation_data=(X_test, X_test)) ```

Answer: return tf.reshape(x,(-1,28,28))

it should be return tf.reshape(decoded,(-1,28,28))
Cause of error: your input x is first passed through a Flatten layer, then through encoder and decoder successively. Finally, call method returns reshaped version of x, but not the decoded. So, there is no possibility to build a computational graph and backpropagate through it.