Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Is it as expected that Polars has slower performance for simple arithmetic operations than Numpy?
Tags: numpy,python-polars

Question: The task I am benchmarking with is simply clipping values element-wise. I have done this using both numpy and polars. But, it turned out that using numpy is much faster (~5x) than using polars (as shown below).
So, my question:

Is this behavior as expected?
If so, does it mean that polars (although it is highly optimized for join/groupby), may not be well-suited for performing relatively simple numerical vector/array operations such as clipping in my example?

import timeit
import numpy as np
import polars as pl

N = 10_000_000
x = np.random.normal(size=N)
y = np.random.normal(size=N)
z = y + 0.5
df = pl.DataFrame({"x": x, "y": y, "z": z})

>>> timeit.timeit(lambda: np.minimum(np.maximum(x, y), z), number=10)
0.60923

>>> timeit.timeit(lambda: df.select(pl.min(pl.max(pl.col("x"), pl.col("y")), pl.col("z"))), number=10)
3.39337

Answer: Update 25th April 2023
As of polars >= 0.17.8 we optimized it a bit more. It is much closer now.
Our horizontal min operation is not really optimized. If you open an issue we can improve that. Most of our optimization attention has gone to the expensive operations.
it depends
To answer your question, no it depends.

Polars can achieve the same numeric performance for columnar operations.
It will be slower for row-wise operations and operations that utilize
blas, like dot products.

In aggregate, polars might be faster; as often, many operations are able to run in parallel. If you feel it is too slow in one area, open an issue.
Title: Python pip error "externally managed environment" after upgrading to Ubuntu 23.04
Tags: python,python-3.x,ubuntu

Question: I'm running Ubuntu Server and upgraded it from 22.10 to 23.04. After the installation finished, I tried running one of my python scripts and one of my modules went missing. I tried to reinstall it with pip3 install playsound but I get this error:
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

As the error says, I tried to install playsound with sudo apt install python3-playsound but it's not available through apt.
I also tried python3 -m pip install playsound and that gives the same error. I don't really know what else to do because I have never needed to mess with pip like this before.
Any thoughts/ideas appreciated!

Answer: Figured it out, just needed to activate the virtual environment I made.
Title: Python pip error "externally managed environment" after upgrading to Ubuntu 23.04
Tags: python,python-3.x,ubuntu

Question: I'm running Ubuntu Server and upgraded it from 22.10 to 23.04. After the installation finished, I tried running one of my python scripts and one of my modules went missing. I tried to reinstall it with pip3 install playsound but I get this error:
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

As the error says, I tried to install playsound with sudo apt install python3-playsound but it's not available through apt.
I also tried python3 -m pip install playsound and that gives the same error. I don't really know what else to do because I have never needed to mess with pip like this before.
Any thoughts/ideas appreciated!

Answer: Use pipx:
install pipx with sudo apt install pipx
then install the package: pipx install playsound
Title: ImportError: cannot import name 'json_normalize' from 'pandas.io.json'
Tags: python,pandas,python-3.9

Question: python 3.9.2-3
pandas 2.0.0
pandas-io           0.0.1

Error: 
from pandas.io.json import json_normalize
ImportError: cannot import name 'json_normalize' from 'pandas.io.json' (/home/casaos/.local/lib/python3.9/site-packages/pandas/io/json/__init__.py)

Apparently this was a problem early on in the pre 1x days of pandas, but seems to have resurfaced.
Suggestions?
I'm running a script which was  functional previously, but migrating it to a new host.
It errors out on the line:
from pandas.io.json import json_normalize

and throws the error
ImportError: cannot import name 'json_normalize' from 'pandas.io.json' (/home/casaos/.local/lib/python3.9/site-packages/pandas/io/json/__init__.py)

I've attempted to reinstall pandas ('install' option), remove and reinstall, and 'install --force-reinstall' all performed as root so that the base install  of python3 has it installed as opposed to a single user

Answer: This was indeed the solution simply removing the import line. I'd have liked an attribute to check to determine the pandas version installed easily, but have to settle for a try:except: to determine if the import is needed.
pandas.io.json.json_normalize was deprecated and removed in the newest version. Use pandas.json_normalize. Also, the tutorial you were following is most probably severely outdated. You are on your own now. –
Ξένη Γήινος
Apr 23 at 7:48
Title: Define constraints with or-tools for a carpooling problem
Tags: python,constraints,linear-programming,or-tools

Question: I am having a real life situation where I am trying to solve a school carpooling problem. I thought it would be good to try and use or-tools and learn in the process.
The school is divided into 2 locations, some children go to location 1 and some to location 2 (for the entire week)
So far, I have defined the model:
from ortools.sat.python import cp_model
def main():
  # Data.
  num_parents = 12
  num_children = 20
  num_days = 5
  num_locations = 2
  all_parents = range(num_parents)
  all_children = range(num_children)
  all_days = range(num_days)
  all_locations = range(num_locations)

  # Creates the model.
  model = cp_model.CpModel()
  # Creates ride variables.
  # rides[(n, d, c, l)]: parent 'n' takes child 'c' on day 'd' to location l.
  rides = {}
  for n in all_parents:
    for d in all_days:
      for c in all_children:
        for l in all_locations:
        rides[(n, d, c, l)] = model.NewBoolVar('ride_n%id%ic%il%i' % (n, d, c, l))

Now I am trying to define the constraints. Here they are:

Each parent can take up to 4 kids in a ride (would be cool to have that number adjustable per parent as some of them have bigger cars, but lets assume 4)
Each parent should only drive to one of the locations during a ride
Maximum 2 rides per parent per day (pickup and drop-off, but can't do more than that)
Parent always drives at least one of their children in a ride (at least one, but not necessarily all of them)
Each child either goes to location 1 or to location 2. It never changes.
I am trying to make it fair so that the work is spread across everyone

Could someone please give me some examples how to model these constraints ? Also, I haven't yet thought of how to model the parent-child relationship in constraint #4 or the child-location relationship in constraints #5

Answer: Assuming you have data on parents & kids (which location in the week), then you can define variables as 
Let C[l] be a set of tuple, dictionary or some kind of combination that indicates kids for a location l
Similarly C[n] be a set that of (parents, kids)
Define binary variables:  x[n,c,d] &  y[n,l,d, m] where m = 1| drop, 0 for pick
n,c,l,d represent parents, kids, location & days
Constraints:
(1) sum(x[n,c,d] over c) <= 4(y[n,l,d,0]+y[n,l,d,1]) for all n,l,d
(2) y[n,l,d,1]+y[n,l,d,0] <= 1
(3) sum(y[n,l,d,0]+y[n,l,d,1] over l) <= 2 for all n,d
(4)  1 <= sum(x[n,c,d] over c in C[n]) for all n,d
(5) y[n,l,d,0]+y[n,l,d,1] <= sum(x[n,c,d] over c in C[l])
Fair & balanced will be your objective function. It may be load per parent like
Additional constraints
 v <= sum(y[n,l,d,0]+y[n,l,d,1] over l,d) <= z for all n
where variables v,z represent the min & max of rides over days d.
Then fair will be to reduce or minimize the difference, obj = z-v
Title: Is there a way to put spaces in a slash command discord.py?
Tags: python,discord.py

Question: I am making a discord bot which can get the player count of a game on different platforms. I would like to be able to make it so that there is one command called /count steam and it will return the game's player count on steam. I already have it setup to do that but I don't know how to put a space in the slash command. I see other bots have it so it is like this: /command game <arg> Here is my current code for the slash command (obviously I have all the other bot stuff):
@bot.tree.command(name="count", description = "Gorilla Tag's steam player count")
async def count_steam(interaction: discord.Interaction):
    await interaction.response.send_message(getSteamPlayers())

Whenever I run this, I get the following error:
Traceback (most recent call last):
  File "D:\Gorilla Bot\main.py", line 93, in <module>
    async def count_steam(interaction: discord.Interaction):
  File "C:\Users\Sensor\AppData\Local\Programs\Python\Python39\lib\site-packages\discord\app_commands\tree.py", line 887, in decorator
    command = Command(
  File "C:\Users\Sensor\AppData\Local\Programs\Python\Python39\lib\site-packages\discord\app_commands\commands.py", line 644, in __init__
    self.name: str = validate_name(name)
  File "C:\Users\Sensor\AppData\Local\Programs\Python\Python39\lib\site-packages\discord\app_commands\commands.py", line 191, in validate_name
    raise ValueError(
ValueError: 'count steam' must be between 1-32 characters and contain only lower-case letters, numbers, hyphens, or underscores.

I am using the latest version of discord.py. I am also using these for the discord module imports (if it helps):
import discord
from discord import app_commands
from discord.ext import commands

Answer: You can use underscore ("_"), so you will take this:
/count_steam
Title: Under-sampling leads to poor results for no apparent reason
Tags: python,random-forest,image-segmentation,imbalanced-data

Question: I am using Random Forest for a semantic segmentation task, with 3 classes, which are imbalanced. First, I just trained the algorithms on random subsets containing 20% of all the pixels (else my memory cannot handle training the algorithms), and got IoU and Balanced accuracy scores of 0.83 and 0.91 on my test dataset (comprising 32 images representative of the dataset).
Then, to handle imbalance, I did this on the full training dataset (not the previous subset):
rus = RandomUnderSampler(random_state = 42)
X_resampled, Y_resampled = rus.fit_resample(X, Y)

leading to a training dataset with similar size to the previous one, but this time with balanced classes. But surprisingly, I notice that IoU and balanced accuracy are now equal to 0.78 and 0.84 on my test dataset.
Of course under-sampling can lead to bad performances, but generally it is because the under-sampled dataset is smaller than the original one. Here, the original one is already a subset.
Can someone tell me what could be the cause of these poor performances ?

Answer: Hypothesis 1: Maybe you're just experiencing variance in your model: that is by changing training data, the performance of your model are changing too and by a large margin.
This can be due to the nature of the data, as well the model you're using. To confirm that try to do at least ten training with different random seeds. Then, aggregate the performance and compute the mean as well standard deviation of both IoU and accuracy. The std provides you the variance of your model: the lower the std, the lower the variability is. Well, you can also approximate that with K-fold cross-validation of just one training run.
Hypothesis 2: when you under-sample the problem can became either simpler or more difficult. If simpler maybe you're are overfitting so regularize the model more (check your training and validation metrics). Conversely if harder, you're underfitting: this may explain the lower performance, if so increase the model complexity or train for more.
Title: Under-sampling leads to poor results for no apparent reason
Tags: python,random-forest,image-segmentation,imbalanced-data

Question: I am using Random Forest for a semantic segmentation task, with 3 classes, which are imbalanced. First, I just trained the algorithms on random subsets containing 20% of all the pixels (else my memory cannot handle training the algorithms), and got IoU and Balanced accuracy scores of 0.83 and 0.91 on my test dataset (comprising 32 images representative of the dataset).
Then, to handle imbalance, I did this on the full training dataset (not the previous subset):
rus = RandomUnderSampler(random_state = 42)
X_resampled, Y_resampled = rus.fit_resample(X, Y)

leading to a training dataset with similar size to the previous one, but this time with balanced classes. But surprisingly, I notice that IoU and balanced accuracy are now equal to 0.78 and 0.84 on my test dataset.
Of course under-sampling can lead to bad performances, but generally it is because the under-sampled dataset is smaller than the original one. Here, the original one is already a subset.
Can someone tell me what could be the cause of these poor performances ?

Answer: Your correct in the fact that under-sampling usually results in poor performance.

Random undersampling deletes examples from the majority class and can result in losing information invaluable to a model.

Additionally, your model was trained on the imbalanced data, which could mean the model performs much better on the majority classes (because this constitutes a big portion of the loss). The new dataset has less of those samples for which the performance is better, so the model performs slightly worse.
However, your sample size is way too low to statistically represent the problem well. My guess is that with a bit of extra training on the augmented data, your model will perform just as well with the balanced data.
Title: Why are the SHAP values of a feature with mostly invalid values all positive?
Tags: python,feature-selection,shap,catboost

Question: I have a CatboostRanker (with groups) model trained on a dataset of ~1m rows of ~300 features. Three such features are ~90% invalid (have a special category denoting such). My data is time series based, so the invalid data of these features is all but the most recent data.
For some reason, these three features are amongst the top 5 most important according to their shapley values (the absolute sum of all shap values for a given feature). When looking at the individual shap values for each individual object, ALL of them are positive, meaning they all contribute positively to the target variable of binary [0,1].
I don't understand why the 90% of invalid values for these features all carry with them a positive shapley value, since the invalid category theoretically confers no helpful information. I've read several explanations of shap values, and understand their mathematical basis, but still no closer to an answer. Any suggestions would be much appreciated.
On the other hand, Catboost's own permutation-based get_feature_importance method ranks these variables lowly which seems far more intuitive.
Method: I used shap.TreeExplainer on the trained model, then extract the shap values from the explainer by passing a catboost pool containing my training data.

Answer: SHAP does not determine if your features are valid or not. It determines the importance of those features to your model's output. I had the same confusion when implementing it for some use cases.
If you go back to the game theory origin of SHAP, you can understand it as this : SHAP does not determine if the player (your feature) is a good player (valid feature) but it determines how much it contributes to the game (your output).
If you want to gain understanding on the validity of your features, you should see the model's performance as well (your metrics).
Title: pandas using any(1) has suddenly started giving errors?
Tags: python,python-3.x,pandas

Question: My code was working perfectly, I updated openpyxl now when I try:
data = {'Col1': ['Charges', 'Realized P&L', 'Other Credit & Debit', 'Some Other Value'],
        'Col2': [100, 200, 300, 400],
        'Col3': ['True', False, 'True', 'False']}
df = pd.DataFrame(data)

# keep rows where certain charges etc are present
filtered_df = df[df.isin(["Charges", "Realized P&L", "Other Credit & Debit"]).any(1)]


I get the error:
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
TypeError: any() takes 1 positional argument but 2 were given

I tried:
filtered_df = df[df.isin(["Charges", "Realized P&L", "Other Credit & Debit"]).any()]
# removed 1 from any

UserWarning: Boolean Series key will be reindexed to match DataFrame index.
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/Volumes/coding/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 3751, in __getitem__
    return self._getitem_bool_array(key)
  File "/Volumes/coding/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 3804, in _getitem_bool_array
    key = check_bool_indexer(self.index, key)
  File "/Volumes/coding/venv/lib/python3.8/site-packages/pandas/core/indexing.py", line 2499, in check_bool_indexer
    raise IndexingError(
pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).

Not sure what happened or why it suddenly stopped working.

Answer: The point is that 1 was previously a positional argument and is now a keyword argument, so you have to specify any(axis=1) explicitly.
Title: Removing items from dictionaries using del keyword
Tags: python,dictionary,del

Question: I am a novice when it comes to using Python language. I want to understand why removing an item from a dictionary using the del keyword is impossible when someone references the item using a variable. It seems the del keyword only works if one uses the square bracket format i.e del dictionary[key] why is this so? I have provided a code snippet to help in understanding my question and guidance when it comes to answering the question (don't forget to mention the concept of id).
>>> personal_details = {'f_name': 'Imani', 'l_name': 'Ritcher', 'age': 25}
>>> first_name = personal_details['f_name']
>>> id(first_name)
139769037520176
>>> id(personal_details['f_name'])
139769037520176
>>> del first_name
>>> personal_details
{'f_name': 'Imani', 'l_name': 'Ritcher', 'age': 25}
>>> first_name
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'first_name' is not defined
>>> del personal_details['f_name']
>>> personal_details
{'l_name': 'Ritcher', 'age': 25}
>>>

Answer: TL;TR In Python variables are technically references to objects, rather than the objects themselves.
When you create first_name (first_name = personal_details['f_name']), you are not creating any new object but a reference to already existing object (being the string Imani). Because f_name element in personal_details dict is also a reference to Imani, then after your assignment both first_name and personal_details['f_name'] point to the same object in memory. And that's why id() returns the same value for both variables.
Deleting is opposite. Use of del is not removing any object, but again, it deletes the reference that was created before. But the target object remains unaffected and that's why you can safely remove i.e. first_name and dict element is unaffected (and vice versa) even if they pointed to the same object. Finally, when you remove both references, then Imani become orphan and then it will be garbage collected by Python at some point.
Title: Files inside static folder are not updated on docker
Tags: python,docker,flask

Question: I have a flask project, usually i push it to dockerhub and i run it using docker run dockerhub-image, and after i updated things inside static folder, i stopped and removed the container and also removed the image, after that i repush it again to dockerhub and re-run it, but when visiting it on the web, the files inside static folder does not change at all (other files outside static folder are changed), had no idea to fix this.
I have searched questions related to this issue but couldn't find any of it.
Q: how do fix this issue?

Answer: Docker tags should ideally be immutable. If you're just using docker run <image> rather than docker run <image>:<new tag>, then you're just running the previously pulled, locally cached latest tag. Running an image doesn't automatically pull the newest latest tag
Also, as commented, your browser can cache static web assets
Title: Warning: Selector got both text and root, root is being ignored
Tags: python,web-scraping

Question: I wrote a scrapy spider that was working as normal, but suddenly started getting this warning:
/home/user/github-repos/scrapper/scrapper/env/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored. super().__init__(text=text, type=st, root=root, **kwargs)
Upon further inspection, the section which produces the error is the following
    __slots__ = ["response"]
    selectorlist_cls = SelectorList

    def __init__(self, response=None, text=None, type=None, root=None, **kwargs):
        if response is not None and text is not None:
            raise ValueError(
                f"{self.__class__.__name__}.__init__() received "
                "both response and text"
            )

        st = _st(response, type)

        if text is not None:
            response = _response_from_text(text, st)

        if response is not None:
            text = response.text
            kwargs.setdefault("base_url", response.url)

        self.response = response
        super().__init__(text=text, type=st, root=root, **kwargs)

The warning specifies that the root is being ignored, even though the constructor requires it. This is a class within the scrapy package so it may relate to an update on their behalf.
This is the only part of my code that interacts with selectors:
    def load_item(self, response: TextResponse, app_id, db_id, urls):
        loader = AppLoader(response=response)
        loader.add_value("app_id", app_id)
        loader.add_value("db_id", db_id)
        loader.add_value("url", response.url)
        loader.add_css("game_title", "#appHubAppName::text")
        loader.add_css("publisher", "#game_highlights .dev_row+ .dev_row a::text")
        loader.add_css("developer", "#developers_list a::text")
        loader.add_css("publish_date", ".date::text")
        loader.add_css("tags", "#glanceCtnResponsiveRight a::text")
        loader.add_css(
            "review_count", "#review_type_all+ label .user_reviews_count::text"
        )
        loader.add_css(
            "positive_review_count",
            "#review_type_positive+ label .user_reviews_count::text",
        )
        loader.add_css(
            "negative_review_count",
            "#review_type_negative+ label .user_reviews_count::text",
        )
        loader.add_value("file_urls", urls)

        return loader.load_item()

Answer: I have met this problem was before. And that is because of one element of my xpath miss  Quotation Mark.
Title: Warning: Selector got both text and root, root is being ignored
Tags: python,web-scraping

Question: I wrote a scrapy spider that was working as normal, but suddenly started getting this warning:
/home/user/github-repos/scrapper/scrapper/env/lib/python3.8/site-packages/scrapy/selector/unified.py:83: UserWarning: Selector got both text and root, root is being ignored. super().__init__(text=text, type=st, root=root, **kwargs)
Upon further inspection, the section which produces the error is the following
    __slots__ = ["response"]
    selectorlist_cls = SelectorList

    def __init__(self, response=None, text=None, type=None, root=None, **kwargs):
        if response is not None and text is not None:
            raise ValueError(
                f"{self.__class__.__name__}.__init__() received "
                "both response and text"
            )

        st = _st(response, type)

        if text is not None:
            response = _response_from_text(text, st)

        if response is not None:
            text = response.text
            kwargs.setdefault("base_url", response.url)

        self.response = response
        super().__init__(text=text, type=st, root=root, **kwargs)

The warning specifies that the root is being ignored, even though the constructor requires it. This is a class within the scrapy package so it may relate to an update on their behalf.
This is the only part of my code that interacts with selectors:
    def load_item(self, response: TextResponse, app_id, db_id, urls):
        loader = AppLoader(response=response)
        loader.add_value("app_id", app_id)
        loader.add_value("db_id", db_id)
        loader.add_value("url", response.url)
        loader.add_css("game_title", "#appHubAppName::text")
        loader.add_css("publisher", "#game_highlights .dev_row+ .dev_row a::text")
        loader.add_css("developer", "#developers_list a::text")
        loader.add_css("publish_date", ".date::text")
        loader.add_css("tags", "#glanceCtnResponsiveRight a::text")
        loader.add_css(
            "review_count", "#review_type_all+ label .user_reviews_count::text"
        )
        loader.add_css(
            "positive_review_count",
            "#review_type_positive+ label .user_reviews_count::text",
        )
        loader.add_css(
            "negative_review_count",
            "#review_type_negative+ label .user_reviews_count::text",
        )
        loader.add_value("file_urls", urls)

        return loader.load_item()

Answer: Upgrading to scrapy 2.9.0 solves the problem.
Title: How to specify 2 différents source for a same package when in prod or in dev when using poetry?
Tags: python-poetry

Question: I am using Python / Poetry to manage my virtual environment.
I would like to install django-listing from pypi when in production but use a local version of django-listing when developping, so I wrote this pyproject.toml file :
[tool.poetry]
...

[tool.poetry.dependencies]
python = "^3.11"
Django = "^4.1.7"
...

[tool.poetry.group.prod]
optional = true

[tool.poetry.group.prod.dependencies]
django-listing = "^0.0.15"

[tool.poetry.group.dev]
optional = true

[tool.poetry.group.dev.dependencies]
django-listing = {path = "../django-listing", develop = true}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

When I do :
poetry install --with prod --without dev

It is always the develop version of django-listing that is used : why ?
This is the same thing if I clear caches and if I remove/re-create the environment : how can I proceed ?
I am using Poetry 1.4 and python 3.11

Answer: As answered on github, poetry 1.4 reads all dependencies regardless they are in a group or not. So it is not possible to declare a package twice with 2 different sources.
Title: How can I deal with data creation that depends on response of a microservice?
Tags: python,django,django-rest-framework,microservices

Question: I have two microservices: one is responsible for manage players and the other is responsible for manage authentication/authorization. When I create a new player inside the player microservice I make a call to create the auth data like:
auth_data = create_auth_object(data) # This will make a post in auth microservice

and after it I use the ID return from auth microservice to create players infos like:
data_example = {
    auth_id: auth_data.get("id"), #sometimes auth_data is empty (explained below)
    name: "example",
    real: "example2"
}

create_player(data_example)

but sometimes the auth microservice return a timeout error (some GCP problem) and I cant create the player without the ID from auth microservice, but the auth data has been created in the database even I got the timeout error. So, if I try to create the player again I will receive an error that show the player already exists, but only the auth data exists, the player data was not created.
How is the proper way to deal with It? Tell me if you need more details.

Answer: Both the player service and the auth service must handle this error and make the operation repeatable:
The auth service must accept another identical creation request and return an OK and the ID (but must of course not create another auth data set if it has been created before).
The player service must not try to create a player if there is no ID, but instead either automatically try again to call the auth service (fixed or dynamic/configurable number of attempts) or return an error to the user so the user can manually repeat the operation.
Title: Flask AttributeError: module 'flask.json' has no attribute 'JSONEncoder'
Tags: python,flask

Question: My flask app was working prior to upgrades. I was having trouble with sending email when there was a forgot-reset-password. To try and fix this I recently upgraded some modules for my flask app. The modules that I upgraded with current versions are:

email-validator==2.0.0.post2
Flask==2.3.1
itsdangerous==2.1.2

The Traceback error that I am  getting now is:
Traceback (most recent call last):
File "C:\Users\my_folder\sales\app.py", line 1, in <module>
from product import app
File "C:\Users\my_folder\sales\product\__init__.py", line 56, in <module>
from product.agents.views import agents_bp
File "C:\Users\my_folder\sales\product\agents\views.py", line 7, in <module>
from product.agents.forms import RegistrationForm, LoginForm, UpdateAccountForm, ResetPasswordForm, RequestResetForm
File "C:\Users\my_folder\sales\product\agents\forms.py", line 1, in <module>
from flask_wtf import FlaskForm
File "C:\Users\my_folder\flask_env\lib\site-packages\flask_wtf\__init__.py", line 4, in <module>
from .recaptcha import Recaptcha
File "C:\Users\my_folder\flask_env\lib\site-packages\flask_wtf\recaptcha\__init__.py", line 1, in <module>
from .fields import RecaptchaField
File "C:\Users\my_folder\flask_env\lib\site-packages\flask_wtf\recaptcha\fields.py", line 3, in <module>
from . import widgets
File "C:\Users\my_folder\flask_env\lib\site-packages\flask_wtf\recaptcha\widgets.py", line 6, in <module>
JSONEncoder = json.JSONEncoder
AttributeError: module 'flask.json' has no attribute 'JSONEncoder'

How do I go about fixing this?

Answer: I had this same issue today. For me, updating my version of connexion solved the issue.
Title: How to do ewm_mean in rust polars?
Tags: python,rust,python-polars,rust-polars

Question: In python we can do:
df.with_columns([
   pl.col("myCol").ewm_mean(50)
])

But how do we do the same in rust?  The following doesn't work:
df.with_columns([
    col("myCol").ewm_mean(50)
])

It fails with No method named ewm_mean found in Expr.
While polars is great, the documentation for Rust is awful!  I've tried searching to documentation, and all I can find is polars_arrow::kernels::ewm, but there is no human text in that area of the documentation.  I've also tried looking at the python source to see if I can work down and figure out what it is doing but I can't trace it further back into PyExpr.
(I expect the 50 won't work either - I'm anticipating having to pass some sort of EWMOptions struct in the rust one, and have to calculate alpha rather than com.  The python source that I can understand seems to calculate alpha first before switching to PyExpr)

Answer: Ok, so it turns out the above code will run, but it requires the ewma feature in the cargo.toml
Title: Pandas df.compare() returns the name of a column
Tags: python,pandas,dataframe

Question: I have two dataframes of simlar size. Lets say df1 and df2. For  both the data frames a common column is selected as index. Let's say the name column which is set as index is Id.
When I run the code df1.equals(df2), it returns False. But when I try to compare both of the data frames using df1.compare(df2) only the indexed column name i.e, Id is returned without any values in it.
What should I conclude from this?

Answer: Use assert_frame_equal(df1, df2, check_names=False)
Title: Can i send delete api request directly in browser url?
Tags: python,api,fastapi

Question: I am using Fastapi , and this delete request is there , so for my hosted api or be it on localhost can i send directly delete request in browser like url --> http://api/delete/2 .
It gives me a method not allowed response and similar for update and put api as well,
Can i directly do through this way or not, ?
I was trying to send this delete API request through url , but the json response says method not allowed..

Answer: By default in browser , api requests are taken as "GET" request . To execute any other type of requests use curl or postman .
Title: The task of using itertools
Tags: python,algorithm,task,python-itertools

Question: Put + signs between some digits of 12345678910111213...N to make the sum of M.
The input is two positive integers N and M.
Conclusion: correct examples per line.
Examples:

Input: 5 15.
Conclusion: 1+2+3+4+5=15
Input: 4 46.
Conclusion: 12+34=46
Input: 15 1117614.
Conclusion: 12+3+4567+891+01112131+4+1+5

But the third example won't work, so please tell me what to do.
from itertools import product

def find_expression(n, m):
    nums = [str(i) for i in range(1, n+1)]
    for p in product(['', '+'], repeat=n-1):
        expr = ''
        for i in range(n-1):
            expr += nums[i] + p[i]
        expr += nums[n-1]
        if eval(expr) == m:
            return expr
    return None

n= 15
m= 1117614
expression = find_expression(n, m)
if expression:
    print(expression + '=' + str(m))
else:
    print('No valid expression found.')

print(find_expression(5, 15)) # Output: 1+2+3+4+5
print(find_expression(4, 46)) # Output: 12+34=46
print(find_expression(15, 1117614)) # Output: 12+3+4567+891+01112131+4+1+5

Answer: I believe the issue is that the answer requires 10, 14, and 15 to be split, but your code only allows for the + to be inserted either side of a whole number.
You'll need to change your nums variable to something like ''.join([str(n) for n in range(1,n+1)]) and change your for loop to iterate over the length of the whole string as opposed to the number of numbers (i.e. 1 to 15 has 21 characters, not 15).
Title: Python created ETH address doesn't match Metamask
Tags: python,ethereum,blockchain,metamask

Question: I am working on a project working with the Ethereum blockchain.
Part of the project requires access to my own ETH wallet address.  However, when I run a check on my code, I don't generate my own address that I use for Metamask.  Any help would be appreciated.
from eth_account import Account

acct = Account.create('<my 12 seed words>')

acct.address

The address in acct.address is not mine.  I have copied my seed words exactly, copied from clipboard, and neither returns my wallet address.
Tried to create my ETH wallet address in Python using eth_account library and was unsuccessful

Answer: Use Account.from_mnemonic(mnemonic) instead which should have the same format as Metamask (BIP39).
The function Account.create() does not take mnemonic as an argument, just anything random.
Title: Is it possible to do live css changes on a HTML page via python?
Tags: python,html,css

Question: I'm pretty new to Python and not sure, where to start searching. For a small project I'm trying to figure out if one can manipulate CSS code via python, so that dynamic changes are visible in a browser without refreshing the page? Say you want to change the background of some DIV from green to red depending on a recurring python expression/events after the page has been loaded, like a status indicator that changes over time.
Update: The project is a simple network scanner. As pings have to be handled asynchronous, I have to find a way to update the online status of a host in a table after the page has been loaded. It's a HTML page hosted on my Mac, which I open in a tool, which displays it as a dektop wallpaper.

Answer: Short answer? Sort of.
What you see in a browser is some application (Chrome, Firefox, Safari), parsing an html file that may contain various text, tags, js, css, etc...and making the determination of how to render that content.  In order for the content to update/change, the html must either re-load or have some javascript  running that directly changes the Document Object Model (DOM). But currently,  there is no mechanism for running python inside of a web page.
Now, that being said, what you could do is have some Javascript that listened or polled a server written in Python that emitted content/messages/events that your JS framework (in the webpage) responded to and would make the appropriate changes in your DOM. It's not uncommon and that's how a lot of web pages do what they do: Some 'Backend', could be Python or C# or even JS, that responds to requests from a 'Frontend' e.g. your webpage or browser, and sends content, sometimes as JSON instead of HTML.
Title: Python explain groupby
Tags: python,dataframe,group-by

Question: I'm working in a analysis of data mining. In a the groupby function is used like this:
df.groupby('tshirts')['id'].count()

What does ['id'] really do? I understand that the function are grouping by tshirts, but then the brackets do not know..
Can you explain for me, please? And, if you can give me an example, I appreciate.
Best regard.
pd: df is a dataframe.

Answer: So in the square brackets after groupby() you usually place the column names that you want to apply the function that follows to (in your case count()). So for example in your case, it groups by tshirts and then counts how many times each unique id value appears in the 'id' column.
If your code was something like df.groupby(['tshirts'])['id', 'size'].count() then it would group by tshirts and then apply the count() function to both id and size columns.
Generally the basic template goes like this df.groupby([list of cols to groupby])[list cols to apply function to].function()
If you want to have a different function for each col in list of cols to apply function to try df.groupby([...]).agg('col1': 'count', 'col2': 'sum')
Title: can't retrieve youtube title using yt.title from pytube library
Tags: python,pytube

Question: After running this:
from pytube import YouTube

yt = YouTube('http://youtube.com/watch?v=2lAe1cqCOXo')
print(yt.title)

python version 3.11.3
I receive this error:
raceback (most recent call last):
  File "C:\Users\jay\PycharmProjects\pythonProject\youtube.py", line 4, in <module>
    print(yt.title)
          ^^^^^^^^
  File "C:\Users\jay\PycharmProjects\pythonProject\venv\Lib\site-packages\pytube\__main__.py", line 346, in title
    raise exceptions.PytubeError(
pytube.exceptions.PytubeError: Exception while accessing title of https://youtube.com/watch?v=2lAe1cqCOXo.

I was expecting to receive a title

Answer: Try going to Pytube/__main__.py in line 244 change client = 'ANDRIOD' to client = 'WEB'
this worked for me.
Title: can't retrieve youtube title using yt.title from pytube library
Tags: python,pytube

Question: After running this:
from pytube import YouTube

yt = YouTube('http://youtube.com/watch?v=2lAe1cqCOXo')
print(yt.title)

python version 3.11.3
I receive this error:
raceback (most recent call last):
  File "C:\Users\jay\PycharmProjects\pythonProject\youtube.py", line 4, in <module>
    print(yt.title)
          ^^^^^^^^
  File "C:\Users\jay\PycharmProjects\pythonProject\venv\Lib\site-packages\pytube\__main__.py", line 346, in title
    raise exceptions.PytubeError(
pytube.exceptions.PytubeError: Exception while accessing title of https://youtube.com/watch?v=2lAe1cqCOXo.

I was expecting to receive a title

Answer: I started using yt-dlp plugin.
Title: Click on a text with role button
Tags: python,selenium-webdriver

Question: I am trying to scrape some website which includes "Load more" text. Down you can see the whole html code.
I tried many solutions but nothing works.
I am using Python and Selenium webdriver. Could you please help me to solve this problem?
HTML code:
<td class="MuiTableCell-root-565 MuiTableCell-body-567 MuiTableCell-paddingNone-571" colspan="5">
<span class="MuiButtonBase-root-2050 MuiButton-root-2023 MuiButton-text-2025 MuiButton-textPrimary-2026" tabindex="0" role="button" aria-disabled="false" style="width:100%;padding:16px;text-align:center">
<span class="MuiButton-label-2024"><span>Load mor...</span>
</span>
<span class="MuiTouchRipple-root-2051"></span>
</span>
</td>

Answer: Install Selenium IDE using Chrome web browser
Press Record
Click about
The code will be generated for you.
Should give you what you need as a starter for 10
Title: How to take the cumulative maximum of a column based on another column
Tags: python,pandas,dataframe

Question: I have a DataFrame like this:
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "realization_id": np.repeat([0, 1], 6),
    "sample_size": np.tile([0, 1, 2], 4),
    "num_obs": np.tile(np.repeat([25, 100], 3), 2),
    "accuracy": [0.8, 0.7, 0.8, 0.6, 0.7, 0.5, 0.6, 0.7, 0.8, 0.7, 0.9, 0.7],
    "prob": [0.94, 0.96, 0.95, 0.98, 0.93, 0.92, 0.90, 0.92, 0.95, 0.9, 0.91, 0.92]
})

df["accum_max_prob"] = df.groupby(["realization_id", "num_obs"])["prob"].cummax()

And I want to know how to create a column with this output:
df["desired_accuracy"] = [0.8, 0.7, 0.7, 0.6, 0.6, 0.6, 0.6, 0.7, 0.8, 0.7, 0.9, 0.7]

Each entry of desired_accuracy equals the accuracy value that corresponds to the row where the highest prob has been achieved so far by group (that is why I create accum_max_prob).
So, for example: the first value is 0.8 because there is no data prior to that, but then the next one is 0.7 because the prob of the second row is greater than the first. The third value stays the same, because the third prob is lower than the second one, so it does not update desired_accuracy. For each pair (realization_id, num_obs) the criteria resets.
How can I do it in a vectorized fashion using Pandas?

Answer: Do it like this:
df['desired_accuracy'] = df['accuracy'].where(df['accum_max_prob'].eq(df['prob'])).ffill()
Title: i need to make an image in the canvas in tkinter heres the code but errors are coming
Tags: python,image,tkinter,canvas,tkinter-canvas

Question: c = tkinter.Canvas(width=275,height=300)
so_ramdom = tkinter.PhotoImage("/Users/vikranthracherla/Downloads/download.jpeg")
c.create_image(image=so_ramdom)

the error
Traceback (most recent call last):
  File "d27.py", line 33, in <module>
    c.create_image(image=so_ramdom)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 2325, in create_image
    return self._create('image', args, kw)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk/Tkinter.py", line 2309, in _create
    cnf = args[-1]
IndexError: tuple index out of range

Answer: You forgot to add coordinates x and y
c.create_image(x, y, ..... Also noticed you are not passing in any coordinates onto.
Title: Type conversions inconsistent in Kotlin
Tags: python,kotlin,types

Question: I started coding on python and moved on to Kotlin/android. I'm curious: when you want to convert types in python, it is generally as simple as str(x), but in kotlin - depending on type - it is different:
Integer.toString(x)
x.toString()
there's probably more since I'm new to this, but am I doing this all wrong or is it really actually this inconsistent. I'm trying to re-learn bad habits :-)

Answer: The standard way is to use toString(). All classes have a toString() function. In general, conversions from one class to another in Kotlin are always done by calling a function on that class instance that starts with the word .to.
The reason Integer.toString() exists is that if you're targeting the JVM, then all the Java standard library is available as well. Java has the concept of "primitives" which are not classes so they don't have functions, and so it provides Integer.toString() to be able to convert a primitive integer into a String.
Java has primitive wrapper classes that you pretty much should never touch in Kotlin:

java.lang.Integer
java.lang.Long
java.lang.Float

etc.
Title: Roman to int conversion problem gives an string index out of range error
Tags: python

Question: I have a simple task to convert roman to int. this is the code I have been trying.

def romanToInt(s):
    """
    :type s: str
    :rtype: int
    """
    j=0
    print(len(s))
    for i in range(len(s)):
        k=s[i]
        l=i+1
        if s[i]=="I":
            if s[l]=="V":
                j+=4
            elif s[l]=="X":
                j+=9
        elif s[i]=="X":
            if s[l]=="L":
                j+=40
            elif s[l]=="C":
                j+=90
        elif s[i]=="C":
            if s[l]=="D":
                j+=400
            elif s[l]=="M":
                j+=900
        else:
            match k :
                case "I":
                    j+=1
                case "V":
                    j+=5
                case "X":
                    j+=10
                case "L":
                    j+=50
                case "C":
                    j+=100
                case "D":
                    j+=500
                case "M":
                    j+=1000
    return j

a=input("Enter number")
b=romanToInt(a)
print(b)

this code gives this error .
Enter numberIII
3
Traceback (most recent call last):
  File "C:\Users\admin\Desktop\Dekstop Folder\pycharm\python\Leetcode\Roman to int.py", line 46, in <module>
    b=romanToInt(a)
      ^^^^^^^^^^^^^
  File "C:\Users\admin\Desktop\Dekstop Folder\pycharm\python\Leetcode\Roman to int.py", line 13, in romanToInt
    if s[l]=="V":
       ~^^^
IndexError: string index out of range

Process finished with exit code 1

I dont know why it says String index out of range.
I tried printing the length and it is in the range but i dont know why it does not work. please help.

Answer: As mentioned in the comments, your problem comes when i reaches len(s)-1, meaning l=i+1 is one past the end of the string.
Instead of using s[l], you could use a slice of s[l:l+1].  This will return an empty string when l is out of bounds instead of generating an error.