Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: how to get the unsigned int value of a claripy.ast.bv.BV object in angr?
Tags: python-3.x,angr

Question: import angr
import claripy
# 创建 angr 项目对象
proj = angr.Project('./angr_study/main', load_options={'auto_load_libs': False})

# 设置函数参数
add_addr = proj.loader.find_symbol('add').rebased_addr

state = proj.factory.call_state(addr=add_addr)
state.regs.rdi = claripy.BVV(1234,64)
state.regs.rsi = claripy.BVV(1234,64)

simgr = proj.factory.simgr(state)
simgr.run()

#deadended 保存了每一种分支结束时的状态
if len(simgr.deadended) > 0:
    for state in simgr.deadended:
        print(state.regs.rax)
else:
    print('Error')

the return value of add function is saved in rax.
but the type of state.regs.rax is claripy.ast.bv.BV. I want to use the value of rax as a unsigned int.
I do it with this code:
ret_val = int(('%s'%state.regs.rax)[6:-1],16)

This method works fine.But I don't think it's elegant. I want to know some other methods to convert state.regs.rax to a python int value.

Answer: You can use state.regs.rax._model_concrete.value
Title: forwarding residual block layer by layer, result is wrong
Tags: python,pytorch,resnet

Question: I wrote a pytorch code with residual connection as follow
all_module = []
for i in range(3):
  layer = nn.Sequential(
      nn.Conv1d(n_hidden_channels, n_hidden_channels),
      nn.LeakyReLU(),
      nn.Conv1d(n_hidden_channels, n_hidden_channels),
      nn.LeakyReLU()
      )
  all_module.append(layer)
module_list = nn.ModuleList(all_module)


# method 1
for layer in module_list:
  x = x + layer(x)
print(x)

# method 2
for layer in module_list:
  y = torch.clone(x)
  for m in layer:
    y = m(y)
  x = x + y
print(x)


why the output of method 1 and 2 is different?
Have no idea why this happened.

Answer: Both methods do the same thing. If you run the two methods sequentially, the first method will update the x; therefore, you will have a different result when you run the second method.
If you copy the x before the first method, you will see both methods will create the same results.
Title: IPython: Render HTML text in Colab if a condition is met
Tags: jupyter-notebook,google-colaboratory,ipython

Question: I want to be able to render some HTML text in Colab if a condition is met but it does not work. I have imported HTML from IPython.display and this works...
HTML('<h1>Hello, World</h1>')

But what I really want is this...
if True:
  HTML('<h1>Hello, World</h1>')

I have also tried...
if True:
  IPython.display.display_html('<h1>Hello, World</h1>')

But it didn't work.
Thanks in advance for your help.

Answer: you should use display() to display your page, e.g.  display(HTML('<h1>Hello, World</h1>'))
Title: Automatically switch Python gettext strings with their translations from a .po file
Tags: python,gettext

Question: I have a Python/Django code base in which strings are encapsulated in gettext calls. For legacy reasons, the strings in the Python files have been written in French and the English translations are inside a .po file.
I now wish to make sure the Python files are in English, strings included. I would like to automatically switch the strings so that the English translations from the .po file end up in the Python files (instead of the French strings), while adding the French strings to a new .po file (matching the new "original" English string).
Since I have a lot of strings, doing this manually would be extremely tedious. Is there any tool or library that could facilitate this process?

Answer: There is no such tool.
If you don't write it yourself, you can use POEdit or Emacs as a helper. Both have a function that lets you jump to the source code line of a PO entry so that copy and paste is a little faster.
Title: Running Selenium Chrome with extension - Chrome not reachable
Tags: python,selenium-webdriver,google-chrome-extension,selenium-chromedriver

Question: I'm developing an app with selenium on an ubuntu EC2 instance. Therefore, there are no displays.
To start Selenium I use xvbf. This is what I used to install xvbf and selenium:
sudo apt-get -y update
sudo apt-get install -y unzip xvfb libxi6 libgconf-2-4 default-jdk xdg-utils
sudo snap install chromium

sudo wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
sudo unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/

pip install selenium

Now, I want to open Selenium within python. If I run the following code, I can get source code from google webpage:
from pyvirtualdisplay import Display
from selenium import webdriver
import time 

display = Display(visible=0, size=(800, 600))
display.start()
print("display started")

# now Chrome will run in a virtual display. 
chromeOptions = webdriver.ChromeOptions() 
#chromeOptions.add_experimental_option('prefs',{"extensions.ui.developer_mode": True,})  # Trial for dev
#chromeOptions.add_argument('--no-startup-window') # This blocks running selenium
#chromeOptions.add_argument("--force-dev-mode-highlighting") # Trial for dev
#chromeOptions.add_argument("--system-developer-mode")  # Trial for dev
chromeOptions.add_argument('--start-maximized')
chromeOptions.add_argument("--remote-debugging-port=9222") # If I don't put a port I get an error about some port
#chromeOptions.add_extension('my.crx')  # This blocks selenium
browser = webdriver.Chrome(chrome_options=chromeOptions)
print("Selenium loaded")

browser.get('http://www.google.com')
print("Page loaded")
time.sleep(3)
print(browser.page_source)
browser.stop_client()
browser.quit()
display.stop()

However, as soon as I uncomment the line for the extension, I get an error: Chrome not reachable. I downloaded the extension from a github project where it stays that you must enable dev tools. Therefore I also tried adding the lines commented with "Trial for dev". These lines do not block Selenium initialization (e.g., if I uncomment them and comment the line for extension Selenium works), but neither I see that adding them have any influence on the extension working. I get the same error.
What should I do?
NOTE: I tested in a windows PC with a device, and without using pyvirtualdisplay the extension works and I can get google source code.

Answer: The code was working under Ubuntu showing the window explorer too.
At the end I removed all configuration for webdriver and used firefox instead of chrome since the extension was in Firefox too. You can init firefox with a virtual display with no problem, and after you init firefox you can apply the extension. That worked for me!
Title: Expression that captures all characters up to a group of characters
Tags: python,regex,python-re

Question: I have several alerts coming from a DC server, which have the following pattern:
alert - name risk score - severity - total

The examples of these alerts would be:
A member was added to a security-enabled local group 47 medium 2
A member was added to a security-enabled universal group 47 medium 1
A security-enabled global group was changed 73 high 2
A security-enabled local group was changed 73 high 2
A user account was locked out  47 medium 31
An attempt was made to reset an accounts password  73 high 14
Member added to security-enabled global group  73 high 2
PowerShell Keylogging Script 73 high 23
PowerShell Suspicious Script with Audio Capture Capabilities 47 medium 23
More Than 3 Failed Login Attempts Within 1 Hour  47 medium 6
Over 100 Connection from 10 Diff. IPs 47 medium 234
Over 100 Connections Attempted 73 high 123
Failed Logins Not Followed by Success Within 2 Hours 21 low 8

I've been using the following pattern to capture only the name of the alerts:
^(\D*)

Essentially, this filters out all of the digits, but now have I received a few alerts I hadn't accounted for. These alerts contain digits in them. For example:
More Than 3 Failed Login Attempts Within 1 Hour  47 medium 6
Over 100 Connection from 10 Diff. IPs 47 medium 234
Over 100 Connections Attempted 73 high 123
Failed Logins Not Followed by Success Within 2 Hours 21 low 8

So I need to be able to capture the complete name, otherwise, I'm ending up with:
More than
Over
Over
Failed Logins Not Followed by Success Within

Despite my efforts, I have not been able to capture the desire pattern. This would be the desired output:
A member was added to a security-enabled local group
A member was added to a security-enabled universal group
A security-enabled global group was changed
A security-enabled local group was changed
A user account was locked out 
An attempt was made to reset an accounts password 
PowerShell Keylogging Script 
PowerShell Suspicious Script with Audio Capture Capabilities
More Than 3 Failed Login Attempts Within 1 Hour
Over 100 Connection from 10 Diff. IPs 
Over 100 Connections Attempted
Failed Logins Not Followed by Success Within 2 Hours

Thanks for taking the time to help!

Answer: The following regex should do the trick: .*\b(?= \d* .* \d*$)
The (?=...) syntax is called a lookahead, and it allows us to specify the text that must follow the specified regex. Here, we're essentially looking for anything followed by the pattern: space, number, space, anything, space, number, end of line.
Title: How to insert multiple values into MySQL database using Python Script
Tags: python,mysql,sql

Question: I have the following sample values;
lst = [{'title': 'Guld för Odermatt i schweizisk dubbel', 'summary': '', 'link':      '``https://www.dn.se/sport/guld-for-odermatt-i-schweizisk-dubbel/``', 'topic': ['empty', 'empty', 'empty', 'empty', 'empty', 'empty', 'SamhalleKonflikter', 'empty', 'empty', 'empty']}
, {'title': 'Bengt Hall blir ny tillförordnad vd på Malmö Opera', 'summary': '', 'link': '``https://www.dn.se/kultur/bengt-hall-blir-ny-tillforordnad-vd-pa-malmo-opera/``', 'topic': ['empty', 'empty', 'empty', 'empty', 'empty', 'empty', 'SamhalleKonflikter', 'empty', 'empty', 'empty']}
, {'title': 'Fyra gripna för grova narkotikabrott', 'summary': '', 'link': '``https://www.dn.se/sverige/fyra-gripna-for-grova-narkotikabrott/``', 'topic': ['empty', 'empty', 'empty', 'empty', 'empty', 'empty', 'SamhalleKonflikter', 'empty', 'empty', 'empty']}]

and I tired using the following script to insert them into my database;
# Connect to MySQL server

`cnxn = mysql.connector.connect(`
`host="localhost",`
`user="root",`
`password="password",`
`database="NewsExtractDb"`
`)`

# Create a cursor object
cursor = cnxn.cursor()

sql = "INSERT INTO database (title, summary, link, topic) VALUES (%s, %s, %s, %s)"
params = [(item['title'], item['summary'], item['link'], ', '.join(item['topic'])) for item in lst]
cursor.executemany(sql, params)cnxn.commit()

But I am keep getting this error;
File "C:\Python311\Lib\site-packages\mysql\connector\connection_cext.py", line 616, in cmd_query
raise get_mysql_exception(
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'database (title, summary, link, topic) VALUES ('Guld för Odermatt i schweizisk ' at line 1
I have tired re-write the code with a for loop statement instead of 'executemany';
sql = "INSERT INTO database (title, summary, link, topic) VALUES (%s, %s, %s, %s)"

for item in lst:values = (item['title'], item['summary'], item['link'], ', '.join(item['topic']))cursor.execute(sql, values)

But I still end up getting the same identical error which I cannot fix. Any ideas?

Answer: After "INSERT INTO" statement there should be valid table name. But in your query table name is "database" which is invalid identifier for table name. To insert data into a table, firstly that table must be created in the NewsExtractDb database. You can try this:
use NewsExtractDb;
create table newstable (
title nvarchar(255),
summary nvarchar(255),
link nvarchar(255),
topic nvarchar(500)
);
and change sql query as below:
sql = "INSERT INTO newstable (title, summary, link, topic) VALUES (%s, %s, %s, %s)"
After this your code shoud run without errors.
Title: Difference between sklearn.neural_network and simple Deep Learning by Keras with Sequential and Dese Nodes?
Tags: python,tensorflow,keras,deep-learning,neural-network

Question: Given ,  sklearn.neural_network  and simple Deep Learning by Keras with Sequential and Dese Nodes, are the mathematically same just two API's with computation optimization?
Yes Keras has a Tensor Support and could also liverage GPU and Complex models like CNN and RNN are permissible.
However, are they mathematically same and we will yield same results given same hyper parameter , random state, input data etc ?
Else apart from computational efficiency what maker Keras a better choice ?

Answer: I don't think they will give you the exact same results as the internal implementations for 2 same operations are different even across pytorch and tensorflow.
What makes Keras a better option is the ecosystem. You have the DataLoaders which can load the complex data in batches for you in the desired format, then you have the Tensorboard where you can see the model training, then you have preprocessing functions especially for data augmentations. In TF/Keras, you now even have data augmentation layers, in PyTorch Torchvision provides this in Transforms. Then you have the flexibility, you can define what types of layers in what order you want, what should be the initializer of the layer, do you want batch norm between layers or not, do you want a dropout layer between the layers or not, what should be the activation of hidden layers you can have relu in 1 layer and tanh in other, you can define how your forward pass should exist, blah blah. Then you have the callbacks to customize the training experience, and so on.
Title: Loading .txt files fast into an pandas df
Tags: python,pandas,dataframe,optimization,dask

Question: This is the code i am using at the moment it is working fine and does exactly what i want.
df_list = []

for file_name in reversed (os.listdir(path)):
        df_small = pd.read_csv(os.path.join(path, file_name), delimiter='\t', decimal='.', skiprows=6)
        df_small = df_small.dropna(subset=[df_small.columns[6]])
        df_list.append(df_small)

df= pd.concat(df_list, ignore_index=True)
print(df)

I am looking for ways to make it faster. At the moment i am loading around 5000 files in the df, the resulting df has around 140 000 rows for this process i need around 20 seconds.
(The files all of have the same layout and around the same size ca.7 kb)
So are there any ways to make it even faster?
Would it make sense to switch to something like dask to read the data even faster or is that unnessary

Answer: you are iterating through the files. if the read or load is what’s taking up the majority of the time, this iterative approach is your bottleneck.
you can use a dask bag to distribute the file paths and load them across however many cores you have available.
Title: Polars with FastAPI and docker
Tags: python,dataframe,multithreading,fastapi,python-polars

Question: I have been exploring Polars for my web application. Its been impressive so far, until I hit this issue that has stalled my use of this awesome library.
Usecase:
I read a parquet file into Polars dataframe, use this pl dataframe to serve results for a get request on FastAPI.
@fastApi.get("/polars-test")
async def polars_test():
    polars_df = pl.read_parquet(f"/data/all_area_keys.parquet")
    df = polars_df.limit(3)
    return df.to_dicts()


polars= 0.16.2
pyarrow=9.0.0
fastapi=0.92.0
BaseDockerImage = tiangolo/uvicorn-gunicorn-fastapi:python3.11

When I package it up into docker image  and run the FastAPI app on gunicorn, this get path does not respond. Using the /docs, hitting this end point will just wait for several minutes and the worker terminates, without any errors logged
I am starting to think Polars multithread is not playing well with FastAPI'S concurrency. But I an unable to find related documents to get an understanding. Please help, would absolutely hate to abandon Polars.
Troubleshooting done so far:

The get request works perfectly when I test it locally.
log on to the running docker container and run the above pl commands - it works
Just tried to print the schema of the dataframe - it works. So the dataframe  is created and metadata available. I get this issue only when I run filter or Any transform on the polars dataframe
Created a lazy frame and tried to collect, but no luck
Remove async from the method, no luck
Changed python version from 3.8 to 3.11, no luck
Spcifying the platform to linus/amd64 while running the docker, no luck

Answer: Having recently switched from pandas to polars for a Dash plotly app (flask also), I can definitely confirm the two don't play well together. The speed increase on the DF operations themselves is impressive (all cores enganged, I'm seeing 10-12x increases in some cases) but once I started replacing pandas code with polars in a computations.py that has a few functions returning DFs, basic operations like reading a file, joining DFs have become extremely hard to debug or even execute.
I've spent hours today trying to figure out why a bunch of files weren't being read. I eventually set the parallel argument to 'none' in .read_parquet and that particular block now works. It now stops at a .join. I have dozens of operations that I managed to parallelize with polars but it seems a nightmare to add this to a web app. Doing a test run with the production recommended settings for Dash i.e. gunicorn, redis for cache and celery with different settings for workers and such and I got a lot of leaked semaphore objects to clean up errors and a perfectly working app now has a lot of choke points.
Not sure where to go from here, I'll probably build a FastAPI endpoint that returns JSONs and read those responses into pandas in Dash for plotting.
Anybody else trying to integrate polars into Dash?
Title: How to get an updated version of the function in class when changing input?
Tags: python-3.x,class

Question: I have this fitness function in this class, I have changed the attributes of an instance of this class, however when calling the function I want the returned value to be updated with the modified input, how can I achieve this? (I wanted to get the output of 12)
class ready:
 def __init__(self,x):
    self.x=x
    self.fitness=fit(self.x)
    
def fit(z):
    return z
p=ready(10)   
p.x=12
print(p.fitness)

Answer: You already updated your x value, but you dont print it. You are printing p.fitness.
When you print p.x you will see you changed the value of x.
Or you just change the value of p.fitness.
Title: Getting AttributeError: 'FigureCanvasAgg' object has no attribute 'set_window_title' using IMGAUG python package
Tags: python,object-detection,image-augmentation

Question: I have found this simple code below on some internet page, but I'm getting an error trying to execute it on my laptop. It should display an image with some bounding boxes on it.
The error occurs not in my code, but in ingaug.py, so inside of the package file. Is it some bug in the imgaug package?
I'm using MacOS, installed imgaug with condo from condo-forge channel.
Python code:
import imageio
import imgaug as ia
from imgaug.augmentables.bbs import BoundingBox, BoundingBoxesOnImage
%matplotlib inline
ia.seed(1)

image = imageio.imread("https://upload.wikimedia.org/wikipedia/commons/8/8e/Yellow-headed_caracara_%28Milvago_chimachima%29_on_capybara_%28Hydrochoeris_hydrochaeris%29.JPG")
image = ia.imresize_single_image(image, (298, 447))

bbs = BoundingBoxesOnImage([
    BoundingBox(x1=0.2*447, x2=0.85*447, y1=0.3*298, y2=0.95*298),
    BoundingBox(x1=0.4*447, x2=0.65*447, y1=0.1*298, y2=0.4*298)
], shape=image.shape)

ia.imshow(bbs.draw_on_image(image, size=2))

Error massage:
AttributeError                            Traceback (most recent call last)
Cell In[1], line 15
      8 image = ia.imresize_single_image(image, (298, 447))
     10 bbs = BoundingBoxesOnImage([
     11     BoundingBox(x1=0.2*447, x2=0.85*447, y1=0.3*298, y2=0.95*298),
     12     BoundingBox(x1=0.4*447, x2=0.65*447, y1=0.1*298, y2=0.4*298)
     13 ], shape=image.shape)
---> 15 ia.imshow(bbs.draw_on_image(image, size=2))

File ~/miniconda3/envs/imgaug/lib/python3.11/site-packages/imgaug/imgaug.py:2120, in imshow(image, backend)
   2117 w = max(w, 6)
   2119 fig, ax = plt.subplots(figsize=(w, h), dpi=dpi)
-> 2120 fig.canvas.set_window_title("imgaug.imshow(%s)" % (image.shape,))
   2121 # cmap=gray is automatically only activate for grayscale images
   2122 ax.imshow(image, cmap="gray")

AttributeError: 'FigureCanvasAgg' object has no attribute 'set_window_title'

Answer: Open "imgaug.py" file, change line 2120 to:

fig.canvas.manager.set_window_title("imgaug.imshow(%s)" % (image.shape,))
Title: Getting AttributeError: 'FigureCanvasAgg' object has no attribute 'set_window_title' using IMGAUG python package
Tags: python,object-detection,image-augmentation

Question: I have found this simple code below on some internet page, but I'm getting an error trying to execute it on my laptop. It should display an image with some bounding boxes on it.
The error occurs not in my code, but in ingaug.py, so inside of the package file. Is it some bug in the imgaug package?
I'm using MacOS, installed imgaug with condo from condo-forge channel.
Python code:
import imageio
import imgaug as ia
from imgaug.augmentables.bbs import BoundingBox, BoundingBoxesOnImage
%matplotlib inline
ia.seed(1)

image = imageio.imread("https://upload.wikimedia.org/wikipedia/commons/8/8e/Yellow-headed_caracara_%28Milvago_chimachima%29_on_capybara_%28Hydrochoeris_hydrochaeris%29.JPG")
image = ia.imresize_single_image(image, (298, 447))

bbs = BoundingBoxesOnImage([
    BoundingBox(x1=0.2*447, x2=0.85*447, y1=0.3*298, y2=0.95*298),
    BoundingBox(x1=0.4*447, x2=0.65*447, y1=0.1*298, y2=0.4*298)
], shape=image.shape)

ia.imshow(bbs.draw_on_image(image, size=2))

Error massage:
AttributeError                            Traceback (most recent call last)
Cell In[1], line 15
      8 image = ia.imresize_single_image(image, (298, 447))
     10 bbs = BoundingBoxesOnImage([
     11     BoundingBox(x1=0.2*447, x2=0.85*447, y1=0.3*298, y2=0.95*298),
     12     BoundingBox(x1=0.4*447, x2=0.65*447, y1=0.1*298, y2=0.4*298)
     13 ], shape=image.shape)
---> 15 ia.imshow(bbs.draw_on_image(image, size=2))

File ~/miniconda3/envs/imgaug/lib/python3.11/site-packages/imgaug/imgaug.py:2120, in imshow(image, backend)
   2117 w = max(w, 6)
   2119 fig, ax = plt.subplots(figsize=(w, h), dpi=dpi)
-> 2120 fig.canvas.set_window_title("imgaug.imshow(%s)" % (image.shape,))
   2121 # cmap=gray is automatically only activate for grayscale images
   2122 ax.imshow(image, cmap="gray")

AttributeError: 'FigureCanvasAgg' object has no attribute 'set_window_title'

Answer: Changing matplotlib version to 3.5 works.
Title: Unable to Use MPS to Run Pre-trained Object Detection Model on GPU
Tags: python,macos,cpu,yolo

Question: I am using MAC and trying to run Ultralytics YOLOv8 pre-trained model to detect objects in my project. However, despite trying to use MPS, I am still seeing the CPU being used in torch even after running the Python code. Specifically, the output I see is: "Ultralytics YOLOv8.0.43 🚀 Python-3.9.16 torch-1.13.1 CPU".
I wanted to know if has support for MPS in YOLOv8, and how can use it?

Answer: Try adding "--device mps" as a parameter when running the command line
Title: Configure Entry Text color for only 1 character in Tkinter
Tags: python,tkinter,tkinter-entry

Question: I am making a project in which I created a tkinter window, and in it there is an Entry widget. I need to change the color of the following Entry widget. The following is what I did.
self.input_entry.config(fg="red")


input_entry is the Entry widget.

This changes the color of the entire string. Now, I want to only change the color of a specific character in the string. How do I accomplish this?
I tried getting the specific index of the input entry, but obviously, it didn't work. Do I need to get the text with self.input_entry.get() and then modify it?
I do not know, can someone inform me of a possible method of doing this?

Answer: You cannot change the color of individual characters in an Entry widget. You can use a one-line Text widget instead, which allows you to apply tags to a range of characters, and then apply various attributes (such as color) to those tags.
Title: Polars vs. Pandas: size and speed difference
Tags: python,pandas,dataframe,python-polars

Question: I have a parquet file (~1.5 GB) which I want to process with polars. The resulting dataframe has 250k rows and 10 columns. One column has large chunks of texts in it.
I have just started using polars, because I heard many good things about it. One of which is that it is significantly faster than pandas.
Here is my issue / question:
The preprocessing of the dataframe is rather slow, so I started comparing to pandas. Am I doing something wrong or is polars for this particular use case just slower? If so: is there a way to speed this up?
Here is my code in polars
import polars as pl

df = (pl.scan_parquet("folder/myfile.parquet")
      .filter((pl.col("type")=="Urteil") | (pl.col("type")=="Beschluss"))
      .collect()
     )
df.head()

The entire code takes roughly 1 minute whereas just the filtering part takes around 13 seconds.
My code in pandas:
import pandas as pd 

df = (pd.read_parquet("folder/myfile.parquet")
    .query("type == 'Urteil' | type == 'Beschluss'") )
df.head()

The entire code also takes roughly 1 minute whereas just the querying part takes <1 second.
The dataframe has the following types for the 10 columns:

i64
str
struct[7]
str (for all remaining)

As mentioned: a column "content" stores large texts (1 to 20 pages of text) which I need to preprocess and the store differently I guess.

EDIT: removed the size part of the original post as the comparison was not like for like and does not appear to be related to my question.

Answer: As mentioned: a column "content" stores large texts (1 to 20 pages of text) which I need to preprocess and the store differently I guess.

This is where polars must do much more work than pandas. Polars uses arrow memory format for string data. When you filter your DataFrame all the columns are recreated for where the mask evaluates to true.
That means that all the text bytes in the string columns need to be moved around. Whereas for pandas they can just move the pointers to the python objects around, e.g. a few bytes.
This only hurts if you have really large values as strings. E.g. when you are storing whole webpages for instance. You can speed this up by converting to categoricals.
Title: How to multiply two columns of a Dataframe?
Tags: python,pandas,dataframe,operation

Question: Good afternoon, I am trying to multiply two columns of a dataframe (C). And add the results to a new column. I have tried different methods but none work. The most common error I encounter is:
TypeError: can't multiply sequence by non-int of type 'float'

The columns that a i want to multiply are: H04_PEDRO_MARIN and SS(mg/l). And also i want to create a new column with the results.
C:
                H04_PEDRO_MARIN SS(mg/l)  multiplication
Fecha
26/07/11 14:00         0.000000     80.4        0.000000
26/07/11 15:00         0.000000     76.1        0.000000
26/07/11 16:00         0.000000        0        0.000000
26/07/11 17:00         0.000000        0        0.000000
26/07/11 18:00         0.000000        0        0.000000
...                         ...      ...             ...
12/04/12 10:00      9430.166667    61.18     9430.166667
12/04/12 11:00      9430.166667    60.05     9430.166667
12/04/12 12:00      9430.166667    59.43     9430.166667
12/04/12 14:00      9430.166667    56.98     9430.166667

[11568 rows x 3 columns]

I have tried:
C['multiplicaction'] = C['H04_PEDRO_MARIN'][1])*(C['SS(mg/l)'])


And
cols = ['H04_PEDRO_MARIN','SS(mg/l)']
C['multiplication'] = C[cols].prod(axis=1)

And don´t work
Even i have tried to separate both columns in different dataframes and multiply and don't work again.
Thanks for any solution.

Answer: Check your columns astype:  C.info().
To solve the TypeError: can't multiply sequence by non-int of type float error, convert the string into a floating-point number before multiplying it with a float. If you convert the string "3" to a float before multiplying it with the floating-point number 3.3 , there will be no error.
Title: Jupyter Notebook - underscore.js doesn't seem to be accessible anymore
Tags: javascript,python,jupyter-notebook,underscore.js,jupyter-widget

Question: I'm using a custom widget in Jupyter. After upgrading to a new machine it stopped functioning. Checking the javascript console in the browser window running the notebook I see the error ReferenceError: _ is not defined. Indeed, running the following in a Jupyter cell:
%%js
alert(_)

I get the same error. Doing the exact same command on my other machine functions correctly (it shows the definition of _ as in underscore.js).
The html source of the Jupyter Notebook still shows underscore.js as being specified in the require.config. Note that simple included widgets still function as expected (so it is not an issue with initializing the widget system).
I haven't found anything in the changelogs of ipywidgets or jupyter regarding changes to the use of underscore.js. I know that the widget api has changed recently in ipywidgets8.0, which is why I am still using version 7.7.3.
Does anyone know if this is an expected change of behaviour in how widgets work? Any other ideas of why underscore does not seem to be initialized properly?

Answer: Underscore.js is available in the 6.4.x series e.g. pip install notebook=6.4.12
It broke in your case likely because js assets were delegated to NbClassic starting with 6.5, which I'm guessing doesn't load underscore.js (or doesn't make it available to the notebook frontend). See github.com/jupyter/notebook/pull/6474.
Title: BeautifulSoup dont take html page in cyrcl
Tags: python,parsing,selenium-webdriver,web-scraping,beautifulsoup

Question: I ran into a problem that when parsing, the soup checks the same page every time.
I use it in conjunction with selenium. Selenium opens a new link without problems, but the soup only checks the very first one.
The saddest thing is that I used similar constructs in other code with another site and it works as it should.
from bs4 import BeautifulSoup
from selenium import webdriver


keys_list = []

def start_browser(link):
    profile = 'C:\\Users\\Crazy_MoT\\AppData\\Local\\Google\\Chrome\\User Data\\Default'
    options = webdriver.ChromeOptions()
    try:
        options.add_argument(f"user-data-dir={profile}")
        browser = webdriver.Chrome(options=options)
    except:
        print("Connect to profile... Error\n Opening new profile")
        browser = webdriver.Chrome()
        browser.quit()
        #browser.get(link)
        
    browser.get(link)
    html = browser.page_source
    soup = BeautifulSoup(html, 'html.parser')

    author = soup.find("a", attrs={'data-qa': 'FileViewAuthorBox'}, href=True)
    print(author["href"])

    keywords = soup.find_all("span", class_="_oX66p")
    for keys in keywords:
        keys_list.append(keys.text)
    print(keys_list)


def start(links):
    for link in links:
        start_browser(link)


links = ["https://ru.depositphotos.com/26182475/stock-photo-happy-birthday.html",
         "https://ru.depositphotos.com/39273619/stock-photo-label-with-happy-birthday.html"]
         
start(links)


I want to collect information from different pages of the site.
I get information only from the very first page and then it repeats

Answer: the code works as it should. When checking on other machines, the error did not appear.
Title: Cython: Aliasing function argument name
Tags: python,c,cython

Question: I have a C library that I am writing a Python extension for using Cython that includes this function (declared in the library's header file):
#include <stdio.h>
#include "zlib.h"

int deflate_index_build(FILE *in, off_t span, struct deflate_index **built);

I am attempting to create a Cython extension for this function using:
from posix.types cimport off_t
from libc.stdio cimport FILE

cdef extern from "header.h":
        int deflate_index_build(FILE *in, off_t span, deflate_index **built)

However, the use of in as the name for the first argument of the function causes a syntax error on compilation because in is a Python keyword. I don't want to change the name of this argument because it would have a large impact on the C library. Is there a way to alias the argument name in Cython to avoid this error?

Answer: From @user2357112 in the comment above:

C argument names aren't part of the function signature. I don't think you actually need to put the same argument names in your Cython code as you declared in your .h file. (You don't even need to match argument names between your .h file and the actual function definition - in fact, you don't even need argument names in your .h file at all.)
Title: Python : Stem TOR Controller: SocketError: Socket error: 0x01: General SOCKS server failure
Tags: python,sockets,google-colaboratory,tor,stem

Question: Hello im having hard time to use the tor stem module, it causes error on the with Controller.from_port(port=9050) as controller:I tried to check if my i am running on port 9050 using netstats, the service on the tor is already enabled
tcp        0      0 127.0.0.1:9050          0.0.0.0:*               LISTEN  

Here's my setup
import requests
import socks
import socket
from stem import Signal
from stem.control import Controller

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
session = requests.session()
def renew_tor_ip():
    with Controller.from_port(port=9050) as controller:
        controller.authenticate(password='malakititeko')
        controller.signal(Signal.NEWNYM)
renew_tor_ip()


The traceback error:
SOCKS5Error                               Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/socks.py in connect(self, dest_pair, catch_errors)
    808                 negotiate = self._proxy_negotiators[proxy_type]
--> 809                 negotiate(self, dest_addr, dest_port)
    810             except socket.error as error:

10 frames
SOCKS5Error: 0x01: General SOCKS server failure

During handling of the above exception, another exception occurred:

GeneralProxyError                         Traceback (most recent call last)
GeneralProxyError: Socket error: 0x01: General SOCKS server failure

During handling of the above exception, another exception occurred:

SocketError                               Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/stem/socket.py in _make_socket(self)
    536       return control_socket
    537     except socket.error as exc:
--> 538       raise stem.SocketError(exc)
    539 
    540 

SocketError: Socket error: 0x01: General SOCKS server failure

Answer: You shouldn't try to connect to the Tor controller through the Tor proxy. You should connect to it directly.
Title: Airflow: how to use dag_run argument value as trigger_rule in PythonOperator
Tags: python,airflow,jinja2

Question: I am trying to pass the trigger_rule of a PythonOperator via the config when triggering my DAG.
For example: {"trigger_rule": "all_done"}.
In other words, I want to be able to at runtime choose which trigger_rule to use by using the config when triggering the DAG.
However, it seems that the trigger_rule field in the PythonOperator is not a templated field. I.e. I can not implement it as follows:
PythonOperator(
   task_id='test',
   python_callable=my_func,
   trigger_rule="{{ dag_run.conf.get('trigger_rule', 'all_success') }}"
)

Any clues on how to tackle this problem?

Answer: This is not possible as trigger_rule is a parameter needed by the Scheduler to determine if a task should run; meaning it needs to be present prior to task runtime. Template fields are evaluated/rendered during task execution.
In essence, if a parameter is needed for the Scheduler it cannot be templated.
Title: prettier throws error `Failed to resolve a parser`
Tags: python,visual-studio-code,vscode-extensions,prettier

Question: Prettier throws error "failed to resolve a parser". Prettier is selected in Workspace, User and Python > Workspace, so I'm out of ideas why the error is thrown...
["INFO" - 08:57:18] File Info:
{
  "ignored": false,
  "inferredParser": null
}
["WARN" - 08:57:18] Parser not inferred, trying VS Code language.
["ERROR" - 08:57:18] Failed to resolve a parser, skipping file. If you registered a custom file extension, be sure to configure the parser.

Answer: Did you by chance try to invoke the "Format Document (Forced)" command (prettier.forceFormatDocument) to trigger the formatting? (I would have asked you this with a comment first if only I had the required reputation to do so.)
I have seen the exact same error when I tried to invoke the command on a .cs file today.
Anyways, here's my...
attempted supportive answer
Since this command came from the "Prettier - Code formatter" extension (esbenp.prettier-vscode), it will only use the extension itself for formatting. Invoking it on any unsupported code only produces the log output you posted. (python and csharp are two unsupported ones)
It would seem that you already have some other extension for formatting python. If that extension is working, the regular command "Format Document" (alt-shift-F / cmd-shift-F) should be enough. But even if it doesn't work, it won't produce the same log output you posted.
Chances are, our respective formatter extensions actually worked, but it did not do the specific thing(s) we had in mind.
For my .cs files, I'm still looking at why my long IF statements are not broken down into multiple lines or why it does not format my indentation on the 2nd and 3rd lines of my multi-line IF statements.

"Isn't this what every prettier should do?"

I probably got this idea from my other projects (i.e. Typescript) which have every formatting needs fulfilled.
next step
You should probably ignore the log from prettier and focus on the log from autopep8 or the lacking of such.
Title: How to prevent IDLE from running a Python code prematurely
Tags: python,python-idle

Question: I am quite new in using IDLE, and yes this is a rookie question, but please bear with me. I have this long, complex python code (I will embed below), that I am copying line by line to IDLE. The problem is that IDLE program runs some parts of the code before I am done typing the whole code. This happens when I skip two lines at a certain section of the code. When I copy the same code as is and input it in one of the online Python interpreters, it runs just fine and the output is complete, unlike with IDLE, where it is in bits or incomplete. How do I stop IDLE from running the code early? The code runs at line 25 (counter += 1) after skipping two lines (so that I get back to the initial/default indentation starting with ">>>"). Here is the code:

decimal_parts = []
for num in decimal_numbers:
    decimal_part = str(num).split('.')[1] if '.' in str(num) else '00'
    if len(decimal_part) == 1:
        decimal_part += '0'
    decimal_parts.append(decimal_part)

lst = list(map(int, ",".join(decimal_parts).split(',')))
start_index = 0
counter = 1
grouped_lists = []

for i, num in enumerate(lst):
    if num == 0:
        start_index = i
        print(f"Position: {i + 1}")
        sub_list = lst[start_index:] + lst[:start_index]
        four_index = sub_list.index(0)
        last_digit_list = [num % 10 for num in sub_list[four_index:]]
        print(f"List {counter}: {last_digit_list}")
        grouped_lists.append(last_digit_list)
        counter += 1

matches = {}
for i, sub_list in enumerate(grouped_lists):
    for j, num in enumerate(sub_list):
        if num not in matches:
            matches[num] = [i]
        else:
            matches[num].append(i)

for num, match_indices in matches.items():
    if len(match_indices) > 1:
        print(f"Matches found for number {num} in lists: {match_indices}")```


Irrespective of the code running prematurely, I continued inputting the rest of the code, then after the output was generated, I would continue typing the other sections, however, I don't want bits of outputs from bits of code. I want to be able to type the whole code, and get the whole output, all in once. Anyways, after "counter +=" line, there is another section of the code I still need to type on a new line starting with ">>>", but I never get to this line as the code runs.

Answer: What version of IDLE are you using? You might be needing to update it to the latest version. Also, be wary of indentation and spacing. You can adjust indentation on IDLE's general settings and switch off automatic indentations... hope this helps.
Title: How to prevent IDLE from running a Python code prematurely
Tags: python,python-idle

Question: I am quite new in using IDLE, and yes this is a rookie question, but please bear with me. I have this long, complex python code (I will embed below), that I am copying line by line to IDLE. The problem is that IDLE program runs some parts of the code before I am done typing the whole code. This happens when I skip two lines at a certain section of the code. When I copy the same code as is and input it in one of the online Python interpreters, it runs just fine and the output is complete, unlike with IDLE, where it is in bits or incomplete. How do I stop IDLE from running the code early? The code runs at line 25 (counter += 1) after skipping two lines (so that I get back to the initial/default indentation starting with ">>>"). Here is the code:

decimal_parts = []
for num in decimal_numbers:
    decimal_part = str(num).split('.')[1] if '.' in str(num) else '00'
    if len(decimal_part) == 1:
        decimal_part += '0'
    decimal_parts.append(decimal_part)

lst = list(map(int, ",".join(decimal_parts).split(',')))
start_index = 0
counter = 1
grouped_lists = []

for i, num in enumerate(lst):
    if num == 0:
        start_index = i
        print(f"Position: {i + 1}")
        sub_list = lst[start_index:] + lst[:start_index]
        four_index = sub_list.index(0)
        last_digit_list = [num % 10 for num in sub_list[four_index:]]
        print(f"List {counter}: {last_digit_list}")
        grouped_lists.append(last_digit_list)
        counter += 1

matches = {}
for i, sub_list in enumerate(grouped_lists):
    for j, num in enumerate(sub_list):
        if num not in matches:
            matches[num] = [i]
        else:
            matches[num].append(i)

for num, match_indices in matches.items():
    if len(match_indices) > 1:
        print(f"Matches found for number {num} in lists: {match_indices}")```


Irrespective of the code running prematurely, I continued inputting the rest of the code, then after the output was generated, I would continue typing the other sections, however, I don't want bits of outputs from bits of code. I want to be able to type the whole code, and get the whole output, all in once. Anyways, after "counter +=" line, there is another section of the code I still need to type on a new line starting with ">>>", but I never get to this line as the code runs.

Answer: The IDLE shell is just an interpreter, you can send multi-line blocks of code at once by using Shift + Enter when starting a new line, but instead it is recommended to run a Python file instead. To do this you make a new file by pressing Control + N or going to File -> New file. Now copy all text into that new file and run it by pressing F5 or by going to Run -> Run module. It may ask you to save your file first, if it does, press Ok and select a location.
Title: Accessing HTML table row element from pandas dataframe
Tags: javascript,html,python-3.x,pandas

Question: I have the following code to generate html from pandas dataframe. I'm using JS to access each table row but getting an error.
  File "<fstring>", line 2
    var elem = array[i].cells[1].innerHTML;
           ^
SyntaxError: invalid syntax


def generate_html_main_page(dataframe: pd.DataFrame):
    # get the table HTML from the dataframe
    table_html = dataframe.to_html(table_id="table")
    # construct the complete HTML with jQuery Data tables
    html = f"""
    <html>
    <header>
        <link href="https://cdn.datatables.net/1.11.5/css/jquery.dataTables.min.css" rel="stylesheet">
    </header>
    <body>
    {table_html}
    <script src="https://code.jquery.com/jquery-3.6.0.slim.min.js" integrity="sha256-u7e5khyithlIdTpu22PHhENmPcRdFiHRjhAuHcs05RI=" crossorigin="anonymous"></script>
    <script type="text/javascript" src="https://cdn.datatables.net/1.11.5/js/jquery.dataTables.min.js"></script>
    <script type="text/javascript">
    var array = document.getElementById("table").rows
    for (let i = 0; i < array.length; ++i) {
            var elem = array[i].cells[1].innerHTML;
            document.getElementById("table").rows[i].cells[1].innerHTML = "<a href='#test'>" + elem +"</a>"
            document.write(document.getElementById("table").rows[i].cells[1].innerHTML)
    }
    </script>
    </body>
    </html>
    """
    # return the html
    return html

Any pointers are greatly appreciated. Thanks!

Answer: You're using an f-string to generate HTML, but the HTML also has {/} characters in it. So Python is trying to execute the body of your JavaScript for loop as Python code.
You can escape them as {{/}} when you want them to just be literal braces. Or use plain string concatenation (html = '''...''' + table_html + '''...''').
Title: Jupyter Notebooks incorrectly calculating numpy conjugation to a power
Tags: python,numpy,jupyter-notebook,jupyter,jupyter-lab

Question: I happened upon some error in my Jupyter Notebook that I can't explain. I have the following code.
import numpy as np
c = 100
c_conj = np.conjugate(c)
print(c == c_conj)
print(c**5 == c_conj**5)

Resulting in the output
True
False

I get the same result for JupyterLite (the online Jupyter Notebook software). Alternatively, if I run the same code on any other platform (e.g. Google Collab), I get the output
True
True

Is this user error? Is there a way to explain this?

Answer: numpy.conjugate(100) returns a numpy.int_ instance, not a plain Python int. numpy.int_ corresponds to C long. On the platforms where the comparison evaluated to False, C long is 32 bits, and the c_conj**5 computation overflowed.
Title: tkinter scrollable listbox refresh
Tags: python,user-interface,tkinter

Question: I have the folling tkinter UI that I am building that on load does not immediately load the listbox data, and I'm not sure why. Instead, on load I get the scrollbar and an empty listbox (the button shows up fine too). As soon as I interact with the window at all, the contents of the listbox show up:
from tkinter import *

gui = Tk()
gui.eval('tk::PlaceWindow . center')
# gui.geometry("500x200")

top_frame = Frame(gui)
top_frame.pack(side=TOP)

bot_frame = Frame(gui)
bot_frame.pack(side=BOTTOM)

scrollbar = Scrollbar(top_frame)
scrollbar.pack(side=LEFT, fill=Y)

lb = Listbox(top_frame)
lb.pack()

def onselect(evt):
    w = evt.widget
    index = int(w.curselection()[0])
    value = w.get(index)
    print(index, value)

lb.bind('<<ListboxSelect>>', onselect)

lb.insert(0, *range(100))

scrollbar.config(command=lb.yview)
lb.config(yscrollcommand=scrollbar.set)

quit_button = Button(bot_frame, text="Quit", command=gui.destroy)
quit_button.pack()

mainloop()

It seems like there is some ordering in which the pack calls need to occur that I can't seem to get right.
How can I get the items to show up on window load while keeping the scrollbar on the left?
EDIT 1: system info:
platform.platform(): macOS-12.6.2-x86_64-i386-64bit
platform.python_version(): 3.10.6
tk.TkVersion: 8.6

Answer: sometimes especially at initiation of the application there is a flood of things tkinter does in the background and pending calls get in the waiting queue/waiting for the mainloop. I've read that many TCL'er are waiting for the application to be mapped and after that forcing it to redraw is a widespread technique to make sure things get considered.
widget.update_idletasks() should do that for you.