Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Locating a Web Element in a Drop-down list by Selenium Python
Tags: python,selenium-webdriver,drop-down-menu

Question: Selenium Python Locator
I use Selenium in Python to to some auto-clicking task for me. The thing is I got this drop-down list, which do not let me send any keys, only select 1 of the option from the list.
I tried to use browser.find_element to search for it by the option I want which named "Deplacement_X1(5)" by 7 locators(id,name,xpath,link test, partial link text, tag name, class name, css selector) but it didn't work.
Full of the option's element is :
<li class="MuiButtonBase-root MuiMenuItem-root MuiMenuItem-gutters MuiMenuItem-root MuiMenuItem-gutters css-18wioi6" tabindex="0" role="option" aria-selected="false" data-value="175155">Deplacement_X1(5)
<span class="MuiTouchRipple-root css-w0pj6f"></span>
</li>

I'm not a web dev, can anyone give me an instruction or a command to locate this?

Answer: Consider the following approaches:

Selenium's recording feature which would capture your input and turn it to selectors
Using the :nth-child(n) css selector
Title: Create a set from a list using `set` vs. unpacking into curly brackets
Tags: python,list,set,unpack

Question: In Python, a set can be created from a list using either the set constructor or by unpacking the list into curly brackets. For example:
my_list = [1, 2, 3]
my_set = set(my_list)

or
my_list = [1, 2, 3]
my_set = {*my_list}

Are there any specific reasons or use cases where one approach is preferred over the other? What are the advantages or disadvantages of each method in terms of performance, readability, or any other relevant factors?

Answer: There is a subtle difference.
set(my_list) produces whatever the callable bound to set returns. set is a built-in name for the set type, but it's possible to shadow the name with a global or local variable.
{*my_list}, on the other hand, always creates a new set instance. It's not possible to change what the brace syntax means without modifying the Python implementation itself (and then, you are no longer implementing Python, but a very Python-like language).
In CPython, using {*mylist} also avoids a function call, it uses BUILD_SET and SET_UPDATE opcodes rather than calling whatever set is bound to.
Title: Pandas to csv method in append mode with dataframes with different columns
Tags: python,pandas

Question: I have a very large number of csv files that I need to merge into a single one. I can't make a list and concatenate later because of memory restrictions even tho I have 64 GB of RAM.
To avoid saving everything into memory, I'm streaming the data into a file using:
entidad_csv = folder_entidad / f"{entidad.name.lower()}.csv"
for f in tqdm(files):
    df = descomprime(f)
    df.to_csv(entidad_csv, index=False, mode="a", header=not entidad_csv.exists())

But I'm having problems when a column is missing in one of the files because it appends the rows as is. I don't know before hand which columns are present in each file so I need the merged file to have every column.
Thanks in advance

Answer: You can try making a list of columns based on files read and make it a blank dataframe with header only and then use the blank dataframe to concat with new csv file. It will make a new dataframe with all columns from files.
Title: tkinter counting number of displayed lines
Tags: python,tkinter

Question: import tkinter as tk

out = """Line 1: Lorem ipsum dolor sit amet
Line 2: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Viverra vitae congue eu consequat ac. Etiam dignissim diam quis enim lobortis scelerisque. Metus dictum at tempor commodo ullamcorper. """

root = tk.Tk()
root.geometry("980x720")
text = tk.Text(root, wrap = "word", font=("TkDefaultFont",18))
text.pack(side="top", fill="x")
text.insert("end", out)


num_lines = text.count("1.0", "end", "displaylines")[0]
print(f"initial num lines: {num_lines}")
text.configure(height=num_lines)

def test(event):
    num_lines = text.count("1.0", "end", "displaylines")[0]
    print("num lines:", num_lines)
    text.configure(height=num_lines)

text.bind('<MouseWheel>', test)

root.mainloop()

I have a tk.Text widget that will be on a scrollable frame, and so should have its height set to the number of lines in the text, depending on the window size so that it does not need to be scrolled.
In this test case, the string is 2 lines but with the initial window size, this takes up 4 lines after wrapping. However, initially the num_lines returns 255 instead of 4 (in console output) so the widget height is not configured correctly.
Only when calling test() by hovering over the text widget and moving the scroll wheel, then the text widget will have its height set correctly depending on the window size.
How can this be set correctly from the start?

Answer: The issue is that the num_lines calculation is being performed before the text widget has been rendered on the screen. As a result, it's returning an incorrect value.
Add root.update() after text.insert("end", out) and you'll be fine.
Hopes that help you.
Title: AttributeError: 'SparkSession' object has no attribute 'write'
Tags: python,databricks

Question: can everyone help me with this one?
import pandas as pd
from pyspark.sql import SparkSession
from pyspark import SparkConf


def sql_writer(db, table_name, df, mode):
  server = 'my-server'
  username = 'username'
  password = 'password'
  schema = 'Reports'
  database = db

  sconf=SparkConf().setAppName("test")
  spark=SparkSession.builder.config(conf=sconf).getOrCreate()

    # Write Spark DataFrame to MSSQL using JDBC
  spark.write \
        .format("jdbc") \
        .option("url", url) \
        .option("dbtable", table_name) \
        .option("user", username) \
        .option("password", password) \
        .mode("append") \
        .save()

  print("DataFrame written to MSSQL successfully!")

Howerver, when I try to run this one:
sql_writer('DatabaseName',
          'TableName',
          mydf,
          mode=mode
        )

Here's what I got:
AttributeError: 'SparkSession' object has no attribute 'write'

Answer: The .write method is called on the Dataframe instance, not on the SparkSession instance - you need to produce a dataframe first before you can write it.  Replace the spark with df, and it will be written
Title: How to make django-filter depend on another django-filter
Tags: python,django,django-filter

Question: I'm using package django-filter and i have some fields which i want to be depend on another field like i have field name and field car and if i choose name Michael in name filter, filter car will show me only cars that Michael has
This looks like a big problem and i dont know how to solve that
filters.py
import django_filters

from django_filters import DateTimeFromToRangeFilter

from django.db.models import Q

from common.filters import CustomDateRangeWidget
from common.models import CounterParty, ObjectList

from .models import OpenPointList


class OpenPointFilter(django_filters.FilterSet):
    """
    Django-filter class to filter OpenPointList model by date, CounterParty name and Object name
    """

    CHOICES = (
        ('Closed', 'Closed'),
        ('Open', 'Open')
    )

    status = django_filters.ChoiceFilter(choices=CHOICES)


    category_choice = django_filters.ModelChoiceFilter(label='Категория',
                                                       queryset=OpenPointList.objects.values_list('category',
                                                                                                  flat=True).distinct())

    category = django_filters.CharFilter(method='custom_category_filter')

    class Meta:
        model = OpenPointList
        fields = (
            'open_point_object',
            'status',
            'category',
        )

Answer: You have to use Ajax for this, like when you select one dropdown then you have to call Ajax and in response, you have to set options in another dropdown and so on.
Title: How do I resolve the 'Stream connection lost' error when using RabbitMQ in Python?
Tags: python,rabbitmq,message-queue

Question: hi guys i have problem with connection.channel() in python for rabbit-mq . when i run my code in send and receive connection close forcibly , this is the error
pika.exceptions.StreamLostError: Stream connection lost: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)

how can i fix this . please help me step by step . thanks
i see this problem in other questions and answers really does not help me , i install rabbitmq over and over i run rabbitmq server and enable management but connection always forcibly closed . i think its my os problem it's windows 10 .  can fix this or should i use linux or docker ?

Answer: I ran into the similar issue while attempting to run a simple RabbitMQ producer locally. Because my producer function is running on another machine, I discovered something relating to version. It worked perfectly once I adjusted the erlang version from Erlang 26 to Erlang 25.3.2.
Title: NeoVim ugly text next to variable assignment
Tags: python,neovim,neovim-plugin

Question: Using NeoVim, I created a Python file for a new project, and noticed that now when I declare a new variable, and go back to normal mode, a text appears looking like this:
a = 1 : Literal[1]
b = 'hello' : Literal['hello']
c = ['a', 'random', 'list'] : list[str]

I am not sure it comes from that, but I use coc-nvim for autocompletion.
Where does this text come from?

Answer: Those are inferred type hints.
If you want to deactivate them this is what you can do:

Go to your coc-settings.json
Add "inlayHint.enable": false

Check :h coc-inlayHint and :h coc-config-inlayHint for more
Title: No module named 'flask' in virtualenv
Tags: python,flask,centos,virtualenv,pgadmin-4

Question: I was running pgAdmin4 in Centos 8 with no problems. I upgraded to pgAdmin4 7.11 and now it no longer works. This is the error:
ModuleNotFoundError: No module named 'flask'

Flask is installed in the virtualenv:
# pwd
/usr/pgadmin4/venv/lib/python/site-packages

# ll -d flask*
drwxrwxr-x. 3 root root  4096 May 29 12:14 flask
drwxrwxr-x. 2 root root    46 May 29 12:14 flask_babel
drwxrwxr-x. 2 root root    98 May 29 12:14 flask_babel-3.1.0.dist-info
drwxrwxr-x. 2 root root    69 May 29 12:14 flask_compress
drwxrwxr-x. 2 root root    43 May 29 12:14 flask_gravatar
drwxrwxr-x. 2 root root   159 May 29 12:14 flask_login
-rw-rw-r--. 1 root root 17950 May  3 11:10 flask_mail.py
drwxrwxr-x. 3 root root    56 May 29 12:14 flask_migrate
drwxrwxr-x. 2 root root    44 May 29 12:14 flask_paranoid
-rw-rw-r--. 1 root root 13860 May  3 11:10 flask_principal.py
drwxrwxr-x. 6 root root  4096 May 29 12:14 flask_security
drwxrwxr-x. 2 root root    67 May 29 12:14 flask_socketio
drwxrwxr-x. 2 root root  4096 May 29 12:14 flask_sqlalchemy
drwxrwxr-x. 3 root root   120 May 29 12:14 flask_wtf

# ll -d Flask*
drwxrwxr-x. 2 root root 147 May 29 12:14 Flask-2.2.5.dist-info
drwxrwxr-x. 2 root root 123 May 29 12:14 Flask_Compress-1.13.dist-info
drwxrwxr-x. 2 root root 148 May 29 12:14 Flask_Gravatar-0.5.0.dist-info
drwxrwxr-x. 2 root root 119 May 29 12:14 Flask_Login-0.6.2.dist-info
drwxrwxr-x. 2 root root 119 May 29 12:14 Flask_Mail-0.9.1.dist-info
drwxrwxr-x. 2 root root 119 May 29 12:14 Flask_Migrate-4.0.4.dist-info
drwxrwxr-x. 2 root root 119 May 29 12:14 Flask_Paranoid-0.3.0.dist-info
drwxrwxr-x. 2 root root  87 May 29 12:14 Flask_Principal-0.4.0.dist-info
drwxrwxr-x. 2 root root 134 May 29 12:14 Flask_Security_Too-5.1.2.dist-info
drwxrwxr-x. 2 root root 119 May 29 12:14 Flask_SocketIO-5.3.4.dist-info
drwxrwxr-x. 3 root root  99 May 29 12:14 Flask_SQLAlchemy-3.0.3.dist-info
drwxrwxr-x. 2 root root 123 May 29 12:14 Flask_WTF-1.1.1.dist-info


$PYTHONPATH is empty:
# source bin/activate
(venv) [root@ck venv]# echo $PYTHONPATH


The virtualenv is using its own python as expected:
(venv) [root@ck venv]# which python
/usr/pgadmin4/venv/bin/python

If I import flask from the virtualenv python interactive it works:
(venv) [root@ck venv]# python
Python 3.9.16 (main, Jan 17 2023, 18:53:15) 
[GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>>

Answer: I found the problem. The wsgi version was built with an earlier Python version different from the one in the virtualenv. I removed wsgi and installed one with the same Python version in the virtualenv and now it works.
Title: Pandas assign with comprehension giving unexpected results
Tags: python,pandas,dataframe

Question: I want to use assign in pandas passing a dictionary comprehension.
Reproducible example:

import pandas as pd

df = pd.DataFrame({
    "a": [1, 2, 3],
    "b": [4, 5, 6],
    "weight": [0.1, 0.2, 0.3]
})

metrics = ["a", "b"]

df = df.assign(
    **{
        f"weighted_{metric}": lambda df: df[metric] * df["weight"]
        for metric in metrics
    }
)
print(df)

Results:
   a  b  weight  weighted_a  weighted_b
0  1  4     0.1         0.4         0.4
1  2  5     0.2         1.0         1.0
2  3  6     0.3         1.8         1.8

I don't get the expected results, for a, I should get 0.1, 0.2, 0.3.
Do you know how can I get weighted_a correctly?

Answer: When using the assignment operator in combination with list comprehension in pandas, it is important to understand how the assignment works in order to avoid unexpected results. In pandas, assignments using list comprehension operate differently than regular assignments.
When you use list comprehension with pandas to create a new column or modify existing columns, it's essential to use the .loc accessor to correctly assign the values. This is because pandas requires explicit indexing when modifying or assigning values to subsets of data.
Title: Why do ROS enviroment break after conda?
Tags: python,linux,docker,conda,ros

Question: I am trying to use Contact-GraspNet in a Docker environment with a base image with pre-installed ROS. There are no issues with my image and I have used it for other purposes for ROS before. The issue is the recommended way of running Contact-Graspnet is through creating the conda enviroment and when I do that my packages don't build anymore and rospy doesn't work. I thought because ros is installed through apt it shouldn't be a problem? Here is the docker image i am using:
... 
general setup of the docker image
...

ENV PATH="/root/miniconda3/bin:$PATH"
RUN apt-get update \
    && apt-get install -y wget

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-py38_23.3.1-0-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-py38_23.3.1-0-Linux-x86_64.sh -b

RUN echo "source activate contact_graspnet_env" >> ~/.bashrc \
    && /bin/bash -c "source ~/.bashrc"

RUN conda --version

WORKDIR /ros-ws

RUN cd src && \
    git clone https://github.com/NVlabs/contact_graspnet.git

RUN cd src/contact_graspnet && conda env create -f contact_graspnet_env.yml

RUN sudo apt-get install freeglut3-dev -y

RUN /bin/bash -c "source activate contact_graspnet_env"

CMD bash

I tried doing a local installation of the conda packages but that didn't allow me to run contact-graspnet and I'm not sure about ROS.

Answer: When you activate a conda environment, it prepends its own directories to the PATH. This can cause issues with ROS.
To fix that we could check that ROS setup scripts are sourced after you activate the conda environment, by adding the command to source the ROS setup script in your .bashrc after the line where you activate the conda environment, the ROS setup script must be something like /opt/ros/melodic/setup.bash.
Title: Exporting data from a .fs file. Plone 3.3.5
Tags: python,plone,zope,zodb

Question: I just got a task saying "See what you can do with this".
I have a 24GB Plone 3.3.5 folder containing a Data.fs file.
No idea what to do with it. No relevant results.
The Plone is 13 years old. Nothing works. Can't be installed on a modern machine.
Docker only runs a Plone 6 installation. I manage to import the Data.fs via
docker cp Data.fs mycontainer:/plone/instance/var/filestorage/Data.fs

But ever since that - the plone container doesn't run. It exits with an error.
I tried installing plone 3.3.5 - another error:
pip install Plone==3.3.5
Collecting Plone==3.3.5
  Using cached Plone-3.3.5.zip (1.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      error in Plone setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected end or semicolon (after version specifier)
          plone.app.contentmenu >= 1.1.6dev-r22380
                                ~~~~~~~~~~~~^
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.


etc. etc. etc.
I had enough.
Does anybody know of an application or some reasonable human way, as an executable program, a phpmyadmin alternative to ZODB, or anything at all, that can just export the data contained in this .fs file?

Answer: You would likely be better off hiring a Plone expert to export the data for you - especially if you are not familiar with Plone, it will save you a lot of time and headache. Even if you set up a unified installer with the correct version, it may not have everything you need to work with the Data.fs, like the add-ons the site had installed.
Also, collective.jsonify will export to JSON from Plone 3, no need to upgrade to Plone 4. But you will need the Plone site working before you can use it.
Title: Why use Zappa in lambda rather than API gateway?
Tags: python,amazon-web-services,zappa

Question: I'm currently working on a project in python. I have seen that there is to main way to create an API (with an SQL base and sqlaclhemy) on AWS : 
1. Use zappa + flask and deploy everything inside one lambda. Zappa handle everything, the major cons is an overload lambda, I thought lambda had to be light as possible, so with this architecture why not use AWS fargate ? 
2. Use api gateway, do one lambda per endpoint, and deploy everything with terrafom. Even if you do one lambda per endpoint, it makes smaller code with less packages and it is easier to maintain.
In both case you can use sqlAlchemy for sql part.
So, my question is simple, why would you chose zappa + flask to deploy an API on lambda ?
I have tried both way.
I can't find why zappa+flask seems to be better. On every tuto they use this architecture why ?

Answer: Use zappa + flask and deploy everything inside one lambda. Zappa handle everything, the major cons is an overload lambda, I thought
lambda had to be light as possible, so with this architecture why not
use AWS fargate ?


AWS Fargate is a completely different service with a completely different billing model. Lambda is billed per number of executions, with a free-tier that could make some smaller projects completely free. Fargate is billed for every second you have the service deployed and running, even if it is just waiting for connections to come in, and Fargate has no free-tier.


Use api gateway, do one lambda per endpoint, and deploy everything with terrafom. Even if you do one lambda per endpoint, it makes smaller code with less packages and it is easier to maintain.
In both case you can use sqlAlchemy for sql part.


If you have the expertise to do this, this is by far the best way to go. Zappa seems primarily designed for people that want to run Django on AWS Lambda, for the cost savings. I think the Flask support was added later, and as you have pointed out it really doesn't give you any benefit unless you want to simply think about coding in Flask and don't want to take the time to learn how to use AWS.
Title: How can I fix 'ImportError: No module named 'mymodule' when installing a Python package using pip?
Tags: python

Question: I'm encountering an issue while trying to install a Python package named 'mymodule'. I followed the standard installation process using pip, but whenever I attempt to import the module in my code, I receive the error message: "ImportError: No module named 'mymodule'."
Here's what I have already tried:
Verified that 'mymodule' is installed by running "pip show mymodule". It shows up as installed with the correct version.
Restarted my Python interpreter and even restarted my computer to ensure a clean environment.
Checked the Python version compatibility. 'mymodule' is listed as compatible with Python 3.6+ and I'm using Python 3.9.
Tried reinstalling the package using "pip uninstall mymodule" followed by "pip install mymodule" to ensure a clean installation.
Confirmed that the package is not a sub-module of another package and should be imported directly.
Despite these efforts, the import error persists. I would greatly appreciate any suggestions or insights on how to resolve this issue and successfully import the 'mymodule' package into my Python environment.

Answer: Check the module name: Ensure that the module name is spelled correctly and matches the case sensitivity. Python is case-sensitive, so 'mymodule' and 'MyModule' would be considered different modules.

Check the module installation: Double-check that the 'mymodule' package was installed successfully. You can use the command pip show mymodule to verify its installation details. If it doesn't appear, reinstall the package using pip install mymodule.

Check the Python environment: Ensure that you are using the correct Python environment where the 'mymodule' package is installed. Sometimes, different Python versions or virtual environments can cause module import issues. Check your Python version and confirm that it matches the requirements of 'mymodule'.

Verify the module's visibility: If the 'mymodule' package is installed in a specific virtual environment, make sure that your Python interpreter is using that environment. Check your IDE or terminal settings to ensure you are using the correct Python environment.

Check the module location: If the 'mymodule' package is not installed in a standard location, you might need to add its installation path to the Python module search path. You can do this by modifying the sys.path list in your code.


If none of these steps resolve the issue, consider providing additional details such as the operating system, Python version, and any relevant code snippets to assist in further troubleshooting.
Title: When resulting in an infeasible model, what tips are there to break down the nexus of the issue
Tags: python,or-tools,mixed-integer-programming,quadratic-programming,cp-sat-solver

Question: I am new to using or-tools and I have been tasked with minimizing a quadratic objective function with lots of IntVar variables. I don't know where to start breaking down what the problem could be concerning an infeasable model and welcome any tips to begin.
I've watched several videos about using or tools and experimented in a test file with a smaller scale but my main problem results in the infeasable model. The videos/experiments I've done have less variables which I am thinking is the issue but, again, this is my first time using OR-tools.
here is the output from when I run the model:
Starting CP-SAT solver v9.6.2534
Parameters: log_search_progress: true enumerate_all_solutions: true
Setting number of workers to 1
Initial optimization model '': (model_fingerprint: 0x8f376cd881ed44f1)
#Variables: 214 (#ints:1 in objective)

126 Booleans in [0,1]
2 in [-100000,100000]
84 in [0,96]
2 in [0,2000000]
#kIntProd: 4004

Starting presolve at 0.00s
Unsat after presolving constraint #1142 (warning, dump might be inconsistent): int_prod { target { vars: 214 coeffs: 3249 offset: 3249 } exprs { vars: 212 coeffs: 570 offset: -102 } exprs { vars: 212 coeffs: 570 offset: -102 } }
Presolve summary:

2 affine relations were detected.
rule 'affine: new relation' was applied 2 times.
rule 'int_prod: divide product by constant factor' was applied 2 times.
rule 'int_prod: linearize product by constant.' was applied 1140 times.
rule 'int_prod: removed constant expressions.' was applied 1140 times.
rule 'int_square: reduced target domain.' was applied 2 times.
rule 'linear: remapped using affine relations' was applied 1 time.
rule 'presolve: iteration' was applied 1 time.
rule 'variables: canonicalize affine domain' was applied 2 times.
Problem closed by presolve.

CpSolverResponse summary:
status: INFEASIBLE
objective: NA
best_bound: NA
integers: 0
booleans: 0
conflicts: 0
branches: 0
propagations: 0
integer_propagations: 0
restarts: 0
lp_iterations: 0
walltime: 0.007945
usertime: 0.007945
deterministic_time: 0
gap_integral: 0

Answer: I only have simple advices:

reduce the model size if this is a parameter
remove all constraints while keeping the problem infeasible
look at the square constraints and the corresponding domains
add assumptions to check the soundness of your data (capacity is >= 0, no crazy ranges of values, ...)
play with the domain of the variables to enlarge them
inject a known feasible solution and try to find where it breaks
Title: Ensuring a minimum time interval between successive observations in a Pandas dataframe
Tags: python,pandas,dataframe,indexing

Question: I have a pandas dataframe which is sorted by a date column. However I wish to ensure a minimum time interval between observations. Say for simplicity this window is 10 minutes, what this means is that if my first observation occurred at 8:05 AM then the second observation must occur at at least 8:15 AM. Any observations occurring between 8:05-8:15 AM must be dropped from the dataframe. Say without loss of generality that after dropping observations the second observation occurs at 8:17 AM. Then all observations between 8:17-8:27 AM are dropped to find the third data point and this process continues.
I have a script which works but iterates over the rows one at a time and is excruciatingly slow as the dataframe has hundreds of thousands of rows. My current script (window is the minimum threshold in minutes):
cur_time = df.iloc[0].Date

for idx, row in df[1:].iterrows():

time_diff = (row.Date - cur_time).total_seconds()

if time_diff > window*60:

    cur_time = row.Date

else:

    trades_df.drop(idx, inplace=True)

Can anyone think of a more speed optimized way of doing this operation? If I switch to the Date column as the index are there functions readily available for performing this function?
Edit: After doing further research the function that I'm looking for is similar to df.resample(window + 'M').first(). However the issue with using this is that my data set is sparsely spaced. I.e. I don't have data for every minute, the gap between data points could be 1 second or it could be 1 month.

Answer: To all future people looking at this thread:
What I found to work for my specific example is:

My date column is given by 'Date'

df.set_index('Date', inplace=True)

window is the minimum threshold in minutes

df.resample(window +'min').first()

Name is a mandatory field in the original dataset

df.dropna('Name', inplace=True)

txTime is a duplicate column I made of the original Date column.
This is because resample puts in new time slots for the observations
but I want to keep the original date

df.reindex('txTime', inplace=True)
Although in my specific case this code creates hundreds of thousands of NA rows from the resample function, it is still lightning fast compared to my original iterrows method.
Title: Altair: showing the value of the current point in the tooltip
Tags: python,pandas,altair,graph-visualization

Question: In the code below, we have a dataset that can be read as: "two cooks cook1, cook2 are doing a competition. They have to make four dishes, each time with two given ingredients ingredient1, ingredient2. A jury has scored the dishes and the grades are stored in _score.
I want to use Altair to show a graph where the x-axis is each dish (1, 2, 3, 4) and the y-axis contains the scores of the two cooks separately. This currently works but the main issue is that on hover, the tooltip does not include the score of the current point that is being hovered.
import altair as alt
import pandas as pd


df = pd.DataFrame({
    "ingredient1": ["potato", "onion", "carrot", "beet"],
    "ingredient2": ["tomato", "pepper", "zucchini", "lettuce"],
    "dish": [1, 2, 3, 4],
    "cook1": ["cook1 dish1", "cook1 dish2", "cook1 dish3", "cook1 dish4"],
    "cook1_score": [0.4, 0.3, 0.7, 0.9],
    "cook2": ["cook2 dish1", "cook2 dish2", "cook2 dish3", "cook2 dish4"],
    "cook2_score": [0.6, 0.2, 0.5, 0.6],
})


value_vars = [c for c in df.columns if c.endswith("_score")]
cook_names = [c.replace("_score", "") for c in value_vars]
id_vars = ["dish", "ingredient1", "ingredient2",] + cook_names
df_melt = df.melt(id_vars=id_vars, value_vars=value_vars,
                  var_name="cook", value_name="score")

chart = alt.Chart(df_melt).mark_circle().encode(
    x=alt.X("dish:O", title="Dish number"),
    y=alt.Y("score:Q", title="Score"),
    color="cook:N",
    tooltip=id_vars
)

chart.show()


I tried explicitly adding the score columns to the tooltip:
    tooltip=id_vars+value_vars

But that yields the following error:

ValueError: cook1_score encoding field is specified without a type; the type cannot be inferred because it does not match any column in the data.

So how can I get altair to also show the score of (only) the currently hovered element?

Answer: cook1_score is not a column in df_melt, which is why you see the error. Setting tooltip=id_vars+['score'] will work.
Title: How to set up Latex in VS Code for Jupyter Notebook (Windows)
Tags: python,visual-studio-code,jupyter-notebook,latex

Question: I have started learning Python and I want to record my sessions through Jupyter Notebooks used in VS Code, as well as to export them to PDF. However, I am unable to do so because it requires Latex and it does not seem to be so easy to install as a regular library such as numpy.
I get the following message when exporting to PDF via VS interface:
[error] If you have not installed xelatex (TeX) you will need to do so before you can export to PDF, for further instructions please look https://nbconvert.readthedocs.io/en/latest/install.html#installing-tex.To avoid installing xelatex (TeX) you might want to try exporting to HTML and using your browsers "Print to PDF" feature.

It also says I'm missing Pandoc although I have installed it via pip. (I believe this has to do with not having TeX). I have checked the link that shows up, but I don't know how to set up Latex in Windows. I have installed MikTex, and a couple extensions in VS (Latex and Latex Workshop). How can  I install it successfully to be able to export to PDF?
When I open Miktex packages, I just get an empty console.
Also, I have used Latex online with Overleaf, and every time I wanted to include a package, I just used \includepackage{...}. Would this work  in the same way in VS? Since it's not online I'm not sure if it would be that easy to download packages.

Answer: In the end I had to install Pandoc not from pip but from the official site, and then add its location to PATH. Now it works.
Title: conda does not update to 23.5.0
Tags: python,anaconda,conda

Question: My conda won't update anymore. Why? What can I do?
While doing a conda update --all on one of my environments, I got the occasional
==> WARNING: A newer version of conda exists. <==
  current version: 23.3.1
  latest version: 23.5.0

Please update conda by running

    $ conda update -n base -c defaults conda

Or to minimize the number of packages updated during conda update use

     conda install conda=23.5.0

So, as usual, I deactivated my current environment an did the conda update -n base -c defaults conda.
Strangely, the latter command runs smoothly, once again telling me that a newer version of conda exists and that I should update and then ... it just terminates with
# All requested packages already installed.

After that, conda is still not updated. When I try a somewhat more forceful approach by doing
conda install -n base -c defaults 'conda>=23.5.0'

my machine goes into outer space for several hours, failing initial frozen solve, retrying flexible solve, analyzing conflicts, etc., seemingly without success ... until I stop the process with CTRL C
I do have (only) two environments, i.e., base and evb. With conda list the former one shows a conda 23.3.1 the latter one a conda 23.5.0. The only environment I keep up to date with conda update --all is the evb one.
So, now I'm stuck. Any suggestions would be welcome.
(B.t.w. I'm also confused why the 'non-base' environment contains a most recent conda. Does conda update --all within an environment also update the conda of that environment? And if so, why care about the conda of the base at all? But anyway, that is not the issue of this post.)

Answer: Answering your own question, likely, is never an optimal practice, but for the record, I eventually found a workaround:

For the purpose of solving the issue, drop the .condarc file momentarily by renaming it.
Activate the base, i.e., conda activate base and do a conda update --all.

This did update the base environment and the conda version contained in base. Following that, I put my .condarc back into place. Subsequent updates of my actual environment for work did not complain about the need to update conda anymore.
So, case closed ... at least until next conda update
Title: "cannot detect OpenGL context" when using Pygame and moderngl?
Tags: python,pygame,python-moderngl

Question: I wrote the simple program below:
import pygame as pg
import moderngl as mgl
import sys

class GraphicsEngine:
    def __init__(self, win_size=(1200,700)):
        pg.init()

        self.WIN_SIZE = win_size
        pg.display.gl_set_attribute(pg.GL_CONTEXT_MAJOR_VERSION, 3)
        pg.display.gl_set_attribute(pg.GL_CONTEXT_MINOR_VERSION, 3)
        pg.display.gl_set_attribute(pg.GL_CONTEXT_PROFILE_MASK, pg.GL_CONTEXT_PROFILE_CORE)

        pg.display.set_mode(self.WIN_SIZE, flags=pg.OPENGL | pg.DOUBLEBUF)

        self.ctx = mgl.create_context()

        self.clock = pg.time.Clock()

    def check_events(self):
        for event in pg.event.get():
            if event.type == pg.QUIT or (event.type == pg.KEYDOWN and event.key == pg.K_ESCAPE):
                pg.quit()
                sys.exit()
    
    def render(self):
        self.ctx.clear(color=(0.08, 0.16, 0.18))

        pg.display.flip()

    def run(self):
        while True:
            self.check_events()
            self.render()
            self.clock.tick(60)
            pg.display.set_caption(pg.time.get_fps())



if __name__ == '__main__':
    app=GraphicsEngine()
    app.run()

and I get this error
File "path\main.py", line 41, in <module>
    app=GraphicsEngine()
        ^^^^^^^^^^^^^^^^
  File "path\main.py", line 16, in __init__
    self.ctx = mgl.create_context()
               ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\AppData\Roaming\Python\Python311\site->packages\moderngl\context.py", line 1968, in create_context
    ctx.mglo, ctx.version_code = >mgl.create_context(glversion=require, mode=mode, **settings)
                                 >^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\\AppData\Roaming\Python\Python311\site->packages\glcontext\__init__.py", line 69, in create
    return wgl.create_context(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: cannot detect OpenGL context

I tried to reinstall moderngl but nothing changed

Answer: I found a solution to my problem. it was some problem with environment variables, it was enough to remove one variable that shouldn't be there and reinstall moderngl. thanks for trying to help.
Title: Do I need any environment variables set to execute some code, call openai's api, and return a response?
Tags: python,api,openai-api,dotenv

Question: I was going through a course in OpenAI's API using an in-browser jupyter notebook page but wanted to copy some example code from there into a local IDE. I installed Python and the jupyter extention in VS Code and the OpenAI library. My code is below:
import openai
import os

# from dotenv import load_dotenv, find_dotenv
# _ = load_dotenv(find_dotenv()) # read local .env file

openai.api_key  = "my api key is here"

def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0, # this is the degree of randomness of the model's output
    )
    return response.choices[0].message["content"]

prompt = f"""
Determine whether each item in the following list of \
topics is a topic in the text below, which
is delimited with triple backticks.

Give your answer as list with 0 or 1 for each topic.\

List of topics: {", ".join(topic_list)}

Text sample: '''{story}'''
"""
response = get_completion(prompt)
print(response)

I installed Python and imported the openai library. When I run I am getting the error:
APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I'm assuming that's because I commented out lines 3 and 4 in the code because I am unsure what they do and do not know how to use the dotenv library. Is it simple to set this up just to make a basic call to the openai API? That's all I'm trying to do with this code right now.

Answer: Usually, you load your API KEY from your .env file but, as you are hardcoding it, you don't need anything else.
The error you are getting might be related to the absence of the topic_list and story definitions.
Title: (python) reading data from a file (mode = 'r') does not work for the second recall
Tags: python,file

Question: Consider the following code:
file = open("DJJ.txt", mode = 'w')    # construct a .txt file with the mode 'w'
file.write("2\n3")                    # write 2 & 3 in this .txt file
file = open("DJJ.txt", mode = 'r')    # read this file
sum = 0
sqsum = 0

# summation                           # sum (2+3)
for u in file: 
    sum += int(u)
print(sum)

# sum of square                       # sum of square (2^2 + 3^2)
for i in file:
    sqsum += int(i)**2
print(sqsum)

The result is
5
0

This code does not calculate sum of square.
The easiest way to correct this is to open the file again:
file = open("DJJ.txt", mode = 'r')
for i in file:
    sqsum += int(i)**2
print(sqsum)

It looks like file is invalid in the second for loop; simply speaking:  this object  can be used only once. If you want to use it twice, you have to open the file again.
My question is why?   and is there any other way to correct this issue?
Thanks!

Answer: You can try to assign the writer object and reader object different names and make sure to close the objects if you want to follow the same format.
Title: xgb.train is working but xgb.cv is not Error: Invalid shape of labels
Tags: python-3.x,xgboost,cross-validation,gridsearchcv

Question: I am trying to run xgb.cv but it is throwing error there are no null values in my dataset and df type is int64. My target class is binary 0 and 1
X_train.shape

returning (431,58)
y_train.shape

returning (431,1)

import xgboost as xgb

dtrain_clf = xgb.DMatrix(X_train, y_train)
dtest_clf = xgb.DMatrix(X_test, y_test)

params = {"objective": "binary:logistic", "tree_method": "hist",'num_class': 2 }
n = 1000

results = xgb.cv(params,
                 dtrain_clf,
                 num_boost_round=n,
                 nfold=10,
                 metrics=["mlogloss", "auc", "merror"]
                )

Error I am getting is:
XGBoostError: [17:58:51] C:\buildkite-agent\builds\buildkite-windows-cpu-autoscaling-group-i-07593ffd91cd9da33-1\xgboost\xgboost-ci-windows\src\objective\regression_obj.cu:52: Check failed: info.labels.Size() == preds.Size() (387 vs. 774) : Invalid shape of labels.
I have almost everything to make this working but it is not.

Answer: Try changing "binary:logistic" to "multi:softprob" keeping "num_class": 2.
Title: Polars memory usage as compared to {data.table}
Tags: python,r,data.table,python-polars

Question: Fairly new to python-polars.
How does it compare to Rs {data.table} package in terms of memory usage?
How does it handle shallow copying?
Is in-place/by reference updating possible/the default?
Are there any recent benchmarks on memory efficiency of the big 4 in-mem data wrangling libs (polars vs data.table vs pandas vs dplyr)?

Answer: How does it handle shallow copying?

Polars memory buffers are reference counted Copy on Write. That means you can never do a full data copy within polars.

Is in-place/by reference updating possible/the default?

No, you must reassign the variable. Under the hood polars' may reuse memory buffers, but that is not visible to the users.

Are there any recent benchmarks on memory efficiency

The question how it relates in memory usage is also not doing respect to design differences. Polars currently is developing an out-of-core engine. This engine doesn't process all data in memory, but will stream data from disk. The design philosophy of that engine is to use as much memory as needed without going OOM. Unused memory, is wasted potential.
Title: How can I auto-format a docstring in PyCharm?
Tags: python,pycharm

Question: I frequently have a situation where I'll write a paragraph in
a docstring for a function, then come back later and want to add to some part of the middle of the paragraph, and that will make me have to re-adjust the rest of the lines to have them be the proper length (not too long, not too short).  This feels like something that should have an automation available for it.
I guess the trickiest part would be that the plugin would need to understand when to take words off of one line and add them to the beginning of the next line.
Is there some plugin or built-in action that will do this for me?

Answer: You can select the docstring and format it using Edit -> Fill paragraph.
Alternatively the only automated tool I can think of (in Pycharm) is the option which automatically wraps the text after the line length reaches certain number of characters; It is available under Code Style settings (Hard wrap at & Wrap on typing). However, this setting is applied on everything (code included), not just docstring.
Title: Indefinite integral of 1/(L-x) dx in sympy python
Tags: python,sympy,calculation,integral,indefinite

Question: I tried to calculate the indefinite integral of 1/(L-x)dx in sympy:
from sympy import *
x, L = symbols('x L')
f = 1/(L-x)
integrate(f, x)

it returned:

-log(-L + x)

while it should be:

-log(L-x)

Is it related to sympy or I missed something, thanks for any explanation.
I have not found an answer for this paricular function.

Answer: Indefinite integrals are defined only up to an additive constant. The difference between log(x - L) and log(L - x) is I*pi which is a constant.
Considering that you might only be interested in real-valued expressions you might prefer one or the other form but which one integrate should return in that case is still ambiguous: the two expressions differ in that one gives real values for x > L and the other gives real values for x < L.
Title: How to add an image to an html file
Tags: javascript,python,html,flask

Question: i have a folder named app where my python script is, in this folder there is another folder named templates where my 3 html scripts are, the image i want to display is on the templates folder, the
structure looks like this:
app/ 

├── app.py

 └── templates/ 

  ├── registro.html 

  ├── registro_exitoso.html

  ├── contact.html 

  └── logo-inmedi.jpg

This is how im displaying the image in the html:
<img src="{{ url_for('static', filename='logo-inmedi.jpg') }}" class="logo">
And in the python script i have this:
app = Flask(__name__) app.static_folder = 'templates/static'
i have tried displaying the image without the static folder, as well as, using jpg and png

Answer: To display images locally in flask, you need to save them in a folder called 'static/images' in the app/ directory. So you can refer to it in the code as <img src="static/images/logo-inmedi.jpg" class="logo">
Title: Python .ipynb file prints output to the terminal without print function .py doesnt
Tags: python,ipynb

Question: I am implementing a sample probability density function. Although there is no print function in my code it prints sample output from my value. I use .ipynb file, and when i copy the code into .py file, it is not printed. I don't understand where is the problem. Here is my code:
import numpy as np

def gen_b():
    
    samples = []
    num_samples = 10000
    
    while len(samples) < num_samples:
        b = np.random.random() * 6
        y = np.random.random()
        if y <= pdf_b(b):
            samples.append(b)
    
    return samples

gen_b()

pdf_b(x) is a simple f(x) = y  function, there is no printing function inside of it.
How can i solve the problem? I don't want to sample output is printed. Here is the output that I don't want to be printed:
[2.140133171853366,
 3.782237422460611,
 4.252458824774285,
 1.9767482772032179,
 2.250855035249897,
 4.788930465436959,
 1.9076515188916072,
 2.745557799097914,
 3.620720920902893,
 2.540311807348857,
 3.3786579434452295,
 4.654430681635432,
 3.885627334171001,
 3.03255430392689,
 2.078660750831954,
 1.563168692059776,
 4.066738029089566,
 4.173320782327262,
 2.3957708748717117,
 2.392456670794856,
 2.0479422380490147,
 3.0472011553416785,
 1.2691569709730193,
 2.930679937209872,
 3.0627475835692044,
...
 2.3660255564569264,
 2.3079729976179957,
 2.8430403284524752,
 3.9573097908844583,
 ...]
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

Answer: IPython is a REPL, a read, evaluate, print loop. The last statement that is evaluated is gen_b(), so the value is printed. If you don't want it printed, one thing you can do is to simply assign the result to something, even a throwaway variable, _ = gen_b() (or just don't run it in a REPL).
Title: Snowflake Python SSO Connection
Tags: python,snowflake-cloud-data-platform

Question: I'm trying to connect to my Snowflake database via python using the authenticator = 'externalbrowser' method, but keep getting the following error:
OperationalError: 250003: Failed to get the response. Hanging? method: post, url: (snowflake url)
I've read a few threads to troubleshoot this issue, but have not made progress.  After running the connection parameters and creating the connection object, I test the connection but the SSO webpage does not automatically open to enable connection.  Any troubleshooting recommendations? Here is the code I'm using:
import snowflake.connector

# Set up the connection parameters
account = 'account name'
user = 'me'
authenticator = 'externalbrowser'
role = 'role'
warehouse = 'wh name'
database = 'db name'
schema = 'sh name'

# Create the connection object
conn = snowflake.connector.connect(
    account=account,
    user=user,
    authenticator=authenticator,
    role=role,
    warehouse=warehouse,
    database=database,
    schema=schema
)

# Test the connection
cur = conn.cursor()
cur.execute('SELECT CURRENT_USER()')
print(cur.fetchone()[0])

# Close the connection
conn.close()

Answer: This problem was related to my connection parameters- I had an extra dash in my account name.  Solution: review connection parameters with more diligence.