Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: LangChain - cannot import langchain.agents.load_tools
Tags: python,python-3.x

Question: I am trying to use LangChain Agents and am unable to import load_tools.
Version: langchain==0.0.27
I tried these:
from langchain.agents import initialize_agent
from langchain.llms import OpenAI
from langchain.agents import load_tools

shows output
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-36-8eb0012265d0> in <module>
      1 from langchain.agents import initialize_agent
      2 from langchain.llms import OpenAI
----> 3 from langchain.agents import load_tools

ImportError: cannot import name 'load_tools' from 'langchain.agents' (C:\ProgramData\Anaconda3\lib\site-packages\langchain\agents\__init__.py)

Answer: I had the same problem using python 3.7.9, I downloaded python 3.10.10 instead and it worked.
Title: Python - clever way to define filenames including string and number suffix, in order to sort them properly later
Tags: python,sorting,filenames

Question: I am looking for a clever way to sort my files in Python.
I am generating many JSON files in a folder which includes a string and index.
Currently I can list them like:
[A_0.json, A_1.json, A_2.json, A_3.json ... A_500.json]

Another folder contains:
[B_0.json, B_1.json, B_2.json, B_3.json ... B_300.json]

In the next step, for each folder, I will run a script to merge all files into one.
So, I would like to keep this naming convention (string + index).
The suffix numbers come from index from Dataframe.
But, I am struggling to merge all JSON files into one with the right sequence of the index. I first sorted files in a folder:
['A_0.json', 'A_1.json', 'A_10.json', 'A_100.json', 'A_101.json', 'A_2.json', 'A_3.json']

What I would like to see is:
['A_0.json', 'A_1.json', 'A_2.json', 'A_3.json', 'A_10.json', 'A_100.json', 'A_101.json'...]

So, the merged file can contain the content of JSON files in the right order. Note: the original JSON files should be preserved. Sorry that this question may mean two questions/steps.
Suggestion to solve this problem is appreciated. If your suggestion requires the slight change of the naming convention, that is not ideal, but I would definitely consider. Many thanks!

Answer: Please try

l = ['A_0.json', 'A_1.json', 'A_10.json', 'A_100.json', 'A_101.json', A_2.json', 'A_99.json']
l.sort(key=len)
print(l)`
Title: How do I write polars dataframe to external database?
Tags: python,sqlite,rust,python-polars

Question: I have big polars dataframe that I want to write into external database (sqlite for example)
How can I do it?
In pandas, you have to_sql() function, but I couldn't find any equivalent in polars

Answer: You can use the DataFrame.write_database method.
Title: What is the fastest way of determining rank of a new element in the existing numpy array without computing ranks for the whole array?
Tags: python,numpy,scipy

Question: Given an array of numbers A and some decimal number X, I would like to know the rank of X in A, and the straightforward way of doing this is to append a new number into the initial array, run rankdata on it and pick the last element, like this:
import numpy as np
from scipy.stats import rankdata

A = np.array([33.25, 40.16, 18.22, 96.34, 71.15, 48.12, 52.41, 83.11, 12.22])
X = 54.17
B = np.append(A, X)
ranks = rankdata(-B) # reverse an array so that the largest value will have rank 1
rank = int(ranks[-1])

Even though it produces the correct result, in order to run it often and on large arrays, it would be useful to obtain it without sorting the whole array.
With that in mind I wonder whether there is a numpy or scipy idiom of doing it faster.

Answer: Since you would like to determine the rank of new items "often and on large arrays", you'll need a sorted list. Hence, you might as well keep the items in a sorted data structure. Some of the data structures with the best complexity for this (O(n log(n))) would be B-trees and heaps.
Title: Delimiter for Splitting each character of a string?
Tags: python,pandas

Question: I am currently trying to split two-character strings into two separate columns for each character in a pandas data.frame, but I've been struggling to find a way to perform the operation on the column without having to iterate through each row.
My starting data.frame looks something like this:
  Initial
0      PT
1      XT
2      ZT

And I'm hoping to split the 'Initial' column into two separate columns containing each character like this:
  S1 S2
0  P  T
1  X  T
2  Z  T

I've used the split() function, and I've tried to find a proper delimiter to supply it which would split every character, but I'm at a loss so far. Is there a good way to do this without needing to iterate over each row?

Answer: Not sure about pandas, but list(text) will create a list of all characters in a string
Then you could try to create a dataframe over a list of lists.
Title: does python allows elif statement without else statement?
Tags: python

Question: While teaching python to a friend i tried this statement :
val = "hi"
if (val=="hello") or ("w" in val):
    print("hello")    
elif(val=="hi"):
    print("hi")  

And to my great surprise it worked. I always tought in Python you couldn't do an elif without else.
Has it been always like that or the syntax has changed since a particular version?

Answer: This has worked in all versions, just without another case you don't consider the case where if and elif conditions are not accepted.
Title: in pyvis I always get this error: "AttributeError: 'NoneType' object has no attribute 'render'"
Tags: python,python-3.x,networkx,attributeerror,pyvis

Question: I want to do a network visualisation using pyvis in the latest version and the python version 3.9.6:
from pyvis.network import Network
g = Network()
g.add_node(0)
g.add_node(1)
g.add_edge(0, 1)
g.show('test.html')

every time I execute g.show() i get this error:
Traceback (most recent call last):
  File "/Users/tom/Library/Mobile Documents/com~apple~CloudDocs/Projekte/Coding_/f1 standings/test2.py", line 3, in <module>
    g.show('nx.html')
  File "/Users/tom/Library/Python/3.9/lib/python/site-packages/pyvis/network.py", line 546, in show
    self.write_html(name, open_browser=False,notebook=True)
  File "/Users/tom/Library/Python/3.9/lib/python/site-packages/pyvis/network.py", line 515, in write_html
    self.html = self.generate_html(notebook=notebook)
  File "/Users/tom/Library/Python/3.9/lib/python/site-packages/pyvis/network.py", line 479, in generate_html
    self.html = template.render(height=height,
AttributeError: 'NoneType' object has no attribute 'render'

I tried updating pyvis, I changed all sorts of details in my code and I imported all of pyvis.network without any results.

Answer: Probably you've installed version 0.3.2.
I had same issue today, downgrading to 0.3.1 helped me
Title: in pyvis I always get this error: "AttributeError: 'NoneType' object has no attribute 'render'"
Tags: python,python-3.x,networkx,attributeerror,pyvis

Question: I want to do a network visualisation using pyvis in the latest version and the python version 3.9.6:
from pyvis.network import Network
g = Network()
g.add_node(0)
g.add_node(1)
g.add_edge(0, 1)
g.show('test.html')

every time I execute g.show() i get this error:
Traceback (most recent call last):
  File "/Users/tom/Library/Mobile Documents/com~apple~CloudDocs/Projekte/Coding_/f1 standings/test2.py", line 3, in <module>
    g.show('nx.html')
  File "/Users/tom/Library/Python/3.9/lib/python/site-packages/pyvis/network.py", line 546, in show
    self.write_html(name, open_browser=False,notebook=True)
  File "/Users/tom/Library/Python/3.9/lib/python/site-packages/pyvis/network.py", line 515, in write_html
    self.html = self.generate_html(notebook=notebook)
  File "/Users/tom/Library/Python/3.9/lib/python/site-packages/pyvis/network.py", line 479, in generate_html
    self.html = template.render(height=height,
AttributeError: 'NoneType' object has no attribute 'render'

I tried updating pyvis, I changed all sorts of details in my code and I imported all of pyvis.network without any results.

Answer: Default self.template = None. You have to set this value using Network().set_template
Title: How to resolve error of Event loop is closed python unittest?
Tags: python,python-asyncio,python-unittest

Question: While implement Python unittest by subclassing IsolatedAsyncioTestCase, it only runs first test case successfully. For any subsequent test case it throws error that event loop is closed. This happens in both Windows and Mac. Could you please suggest how to make sure that event loop is running during the execution of test within each of the subsclasses  of the IsolatedAsyncioTestCase that I have implemented.

Answer: I had the same problem when trying to run integration tests. The first test passed, but the second one got an "Event loop is closed" error. I'm using MognoDB with the async driver. The reason for this error was the way the database connection was opened. IsolatedAsyncioTestCase creates a new event loop at the start and closes it at the end for execution.
So, a driver connection was attached to the event loop of the first TestCase, and when the second TestCase starts, it throws an error because the event loop of the first TestCase is already closed, but the new connection in the new event loop is not created.
The solution is to create a new database connection in each IsolatedAsyncioTestCase.
Title: Variable is not accessed in Pylance
Tags: python,function,variables,pylance

Question: For some reason, I am receiving an error on VSCode that says the specific variable is not accessed Pylance and each variable says the same thing,
Am I missing something simple?
full code below:
score = 0   


def gen1_questions():
    q1 = input("Question 1. What is the rarest M&M color? ")
    q2 = input(
        "Question 2. In a website browser address bar, what does “www” stand for? ")
    q3 = input(
        "Question 3. Which country consumes the most chocolate per capita? ")
    q4 = input("Question 4. Who was the very first American Idol winner? ")
    q5 = input(
        "Question 5. What is the tiny piece at the end of a shoelace called? ")
    q6 = input("Question 6. How many weeks are in a year? ")
    q7 = input("Question 7. Which animal can be seen on the Porsche logo? ")
    q8 = input("Question 8. Muhammad Ali was well-known in which sport? ")
    q9 = input("Question 9. What is the lowest army rank of a US soldier? ")
    q10 = input(
        "Question 10. What is often seen as the smallest unit of memory greater than a byte? ")


def gen2_questions():
    q1 = input(
        "Question 1. Which is one of two U.S. states does not observe Daylight Saving Time? ")
    q2 = input(
        "Question 2. Michael Jordan won how many NBA titles with the Chicago Bulls? ")
    q3 = input("Question 3. What color eyes do most humans have? ")
    q4 = input("Question 4. What is the hardest rock on earth? ")
    q5 = input("Question 5. What is the solar systems hottest planet? ")
    q6 = input("Question 6. What is the fastest-flying bird in the world? ")
    q7 = input(
        "Question 7. Who was the first woman to have four country albums reach No. 1 on the Billboard 200? ")
    q8 = input(
        "Question 8. What is illegal for a single lady to do in Florida solely on Sundays? ")
    q9 = input("Question 9. Which is the Worlds Largest Ocean? ")
    q10 = input(
        "Question 10. What type of exercise is best for getting the blood flowing? ")

Answer: If that's the only code you have then, yes, you set (for example) q1 to a value but never use it after that.
If you've failed to show us code outside of those functions that uses q1, it still is not being used.
That's because, if you assign to a variable (like q1 = 42) within a function and that variable is not explicitly marked global, it will be a new local variable within the function, not one already existing in a containing namespace.
Title: Functions in Python's math module and equivalents in NumPy library: What are the fundamental differences?
Tags: python,numpy

Question: What are the fundamental differences between the functions in Python's math module and their equivalents in the NumPy library?

Answer: numpy works with vectors (or scalars, or matrices, or arbitrary n-dimensional arrays), math works with scalars only.
Title: ImportError: cannot import name 'dtreeviz' from 'dtreeviz.trees' (/usr/local/lib/python3.8/dist-packages/dtreeviz/trees.py)
Tags: python,google-colaboratory

Question: When I try to run this vizulization on google colab I am getting this error,
ImportError: cannot import name 'dtreeviz' from 'dtreeviz.trees' (/usr/local/lib/python3.8/dist-packages/dtreeviz/trees.py)
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from dtreeviz.trees import dtreeviz


rf = RandomForestClassifier(n_estimators=100,
                            max_depth=3,
                            max_features='auto',
                            min_samples_leaf=4,
                            bootstrap=True,
                            n_jobs=-1,
                            random_state=0)
rf.fit(X, y)

viz = dtreeviz(rf.estimators_[99], X, y,
               target_name="SizeClass",
               feature_names=X_train.columns,
               class_names=list(y_train.feature_names),
               title="100th decision tree")

viz.save("decision_tree.svg")

from google.colab import files
files.download("decision_treef.svg")

I tried pip installing but it says that the requirments are already met

Answer: Try installing and old version
pip install dtreeviz==1.4.0
Title: ImportError: cannot import name 'dtreeviz' from 'dtreeviz.trees' (/usr/local/lib/python3.8/dist-packages/dtreeviz/trees.py)
Tags: python,google-colaboratory

Question: When I try to run this vizulization on google colab I am getting this error,
ImportError: cannot import name 'dtreeviz' from 'dtreeviz.trees' (/usr/local/lib/python3.8/dist-packages/dtreeviz/trees.py)
from sklearn.datasets import load_wine
from sklearn.ensemble import RandomForestClassifier
from dtreeviz.trees import dtreeviz


rf = RandomForestClassifier(n_estimators=100,
                            max_depth=3,
                            max_features='auto',
                            min_samples_leaf=4,
                            bootstrap=True,
                            n_jobs=-1,
                            random_state=0)
rf.fit(X, y)

viz = dtreeviz(rf.estimators_[99], X, y,
               target_name="SizeClass",
               feature_names=X_train.columns,
               class_names=list(y_train.feature_names),
               title="100th decision tree")

viz.save("decision_tree.svg")

from google.colab import files
files.download("decision_treef.svg")

I tried pip installing but it says that the requirments are already met

Answer: I had the same problem. After trying both solutions, I can confirm that installing the older version dtreeviz==1.4.0 fixed the problem.
Title: What do I do when I change poetry pyproject.toml?
Tags: python-poetry

Question: I have a pyproject.toml and I already did poetry init (obviously) and poetry install.
If I change the toml file by hand, what exactly do I have to do?
On the one hand I think I have to synchronize the poetry.lock file but do I erase it and do install again?
I have conflicting ideas on how to proceed after an edit of the toml file.
Also, is the procedure the same if I do a poetry add instead of editing the toml manually?

Answer: Whenever you change Poetry related stuff in your pyproject.toml run poetry lock --no-update afterwards to sync the poetry.lock files with those changes. The --no-update flag tries to preserve existing versions of dependencies. Once the lock file is updated run poetry install to sync your venv with the locked dependencies.
Wherever possible you should prefer using Poetry's cli instead of manually edit the pyproject.toml. Poetry will take care of the steps described above for you. So if you run poetry add <somedep>, Poetry will add the entry to your pyproject.toml, update the poetry.lock and will install necessary dependencies.
Title: Placement of Flatten layer in deep learning model
Tags: python,tensorflow,keras,deep-learning,conv-neural-network

Question: I make a deep learning model for classification. The model consist of 4 Conv2d layer, 1 pooling layer, 2 dense layer and 1 flatten layer. When i do this arrangement of layers: Conv2D, Conv2D, Conv2D, Conv2D, pooling, dense, flatten, dense then my results are good. But when i follow this arrangement: Conv2D, Conv2D, Conv2D, Conv2D, pooling, flatten, dense, dense then the classification results are not good. My question is putting flatten layer between two dense layer is correct or not?
Can I follow the pattern of layer by which i am getting good classification results?

Answer: Typically, it is not recommended to sandwich a flatten layer between dense layers, and as suggested by Corralien, It doesn't provide any value. Your other architecture Conv2D, Conv2D, Conv2D, Conv2D, pooling, flatten, dense, dense is more legit. If your model is providing you with good results, you might want to keep it, but technically you do not need the flatten layer between the two dense layers.
You can consider using Conv2D, Conv2D, Conv2D, Conv2D, Pooling, dense, dense. Or a better alternative would be to try playing with your architecture. Such as adding another pooling layer between the four Conv2d layers like: Conv2D, Conv2D, Pooling, Conv2D, Conv2D, Pooling, flatten, dense, dense, and proceed with adjusting your hyperparameters.
Title: What is the input dimension for a LSTM in Keras?
Tags: python,keras,lstm

Question: I'm trying to use deeplearning with LSTM in keras .
I use a number of signal as input (nb_sig) that may vary during the training with a fixed number of samples (nb_sample)
I would like to make parameter identification, so my output layer is the size of my parameter number (nb_param)
so I created my training set of size (nb_sig x nb_sample) and the label (nb_param x nb_sample)
my issue is I cannot find the correct dimension for the deep learning model.
I tried this :
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, LSTM

nb_sample = 500
nb_sig = 100 # number that may change during the training
nb_param = 10


train = np.random.rand(nb_sig,nb_sample)
label = np.random.rand(nb_sig,nb_param)


print(train.shape,label.shape)
DLmodel = Sequential()
DLmodel.add(LSTM(units=nb_sample, return_sequences=True, input_shape =(None,nb_sample), activation='tanh'))
DLmodel.add(Dense(nb_param, activation="linear", kernel_initializer="uniform"))

DLmodel.compile(loss='mean_squared_error', optimizer='RMSprop', metrics=['accuracy', 'mse'], run_eagerly=True)
print(DLmodel.summary())

DLmodel.fit(train, label, epochs=10, batch_size=nb_sig)

but I get this error message:
Traceback (most recent call last):
  File "C:\Users\maxime\Desktop\SESAME\PycharmProjects\LargeScale_2022_09_07\di3.py", line 22, in <module>
    DLmodel.fit(train, label, epochs=10, batch_size=nb_sig)
  File "C:\Python310\lib\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Python310\lib\site-packages\keras\engine\input_spec.py", line 232, in assert_input_compatibility
    raise ValueError(
ValueError: Exception encountered when calling layer "sequential" "                 f"(type Sequential).

Input 0 of layer "lstm" is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: (100, 500)

Call arguments received by layer "sequential" "                 f"(type Sequential):
  • inputs=tf.Tensor(shape=(100, 500), dtype=float32)
  • training=True
  • mask=None

I don't understand what I'm suppose to put as input_shape for the LSTM layer and as the number of signals I use during the training will changed, this is not so clear to me.

Answer: The input to the LSTM should be 3D with the first dimension being the sample size in your case 500. Assuming input having shape (500,x,y), input_shape should (x,y).
Title: Run flow on prefect cloud without running local agent locally?
Tags: python,deployment,prefect

Question: I'm trying to deploy my flow but I'don't know what I should do to completely deploy it (serverless).
I'm using the free tier of Prefect Cloud and I have create a storage and process block.
The step I have done :

Build deployment

$ prefect deployment build -n reporting_ff_dev-deployment flow.py:my_flow


Apply configuration

$ prefect deployment apply <file.yaml>


Create block

from prefect.filesystems import LocalFileSystem
from prefect.infrastructure import Process

#STORAGE
my_storage_block = LocalFileSystem(
    basepath='~/ff_dev'
)
my_storage_block.save(
    name='ff-dev-storage-block',
    overwrite=True)

#INFRA
my_process_infra = Process(
    working_dir='~/_ff_dev_work',
)
my_process_infra.save(
    name='ff-dev-process-infra',
    overwrite=True)


deploy block

$ prefect deployment build -n <name>  -sb <storage_name> -ib <infra_name>  <entry_point.yml> -a

I know that prefect cloud is a control system rather than a storage medium but as I understand,  a store block -> store the code and process code -> run the code. What is the next step to run the flow without local agent ?

Answer: Where are you looking for the code to be executed from?
With a deployment registered, you can execute the following to spawn a flow run. A deployment just describes how and where -
prefect deployment run /my_flow
Title: Open CV ImportError: /lib/x86_64-linux-gnu/libwayland-client.so.0: undefined symbol: ffi_type_uint32, version LIBFFI_BASE_7.0
Tags: python,opencv,conda,undefined-symbol,libffi

Question: I have installed OpenCV and when trying to import cv2 in python, I get the following error. The import was working fine until I installed/un-installed and re-installed tensor flow.
OpenCV has been installed in a conda environment using cmake.
Any idea how to fix this?
Python 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/deleeps/anaconda3/envs/zscore/lib/python3.10/site-packages/cv2/__init__.py", line 102, in <module>
    bootstrap()
  File "/home/deleeps/anaconda3/envs/zscore/lib/python3.10/site-packages/cv2/__init__.py", line 90, in bootstrap
    import cv2
ImportError: /lib/x86_64-linux-gnu/libwayland-client.so.0: undefined symbol: ffi_type_uint32, version LIBFFI_BASE_7.0
>>> 


$ ldconfig -p | grep libwayland-client
    libwayland-client.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libwayland-client.so.0
    libwayland-client.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libwayland-client.so
    libwayland-client++.so.0 (libc6,x86-64) => /lib/x86_64-linux-gnu/libwayland-client++.so.0

Answer: I got around a similar issue by doing:
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libffi.so.7
Title: Why does timestamp() show an extra microsecond compared with subtracting 1970-01-01?
Tags: python,datetime,floating-point,timestamp

Question: The following differ by 1 microsecond :
In [37]: datetime(2514, 5, 30, 1, 53, 4, 986754, tzinfo=dt.timezone.utc) - datetime(1970,1,1, tzinfo=dt.timezone.utc)
Out[37]: datetime.timedelta(days=198841, seconds=6784, microseconds=986754)

In [38]: datetime(2514, 5, 30, 1, 53, 4, 986754, tzinfo=dt.timezone.utc).timestamp()
Out[38]: 17179869184.986755

The number of microseconds in 986754 in the first case, and 986755 in the second.
Is this just Python floating point arithmetic error, or is there something else I'm missing?

Answer: It an error of 1.37109375 usecs due to conversion to a 64 bit float:



print("%100.100f\n" % (17179869184.986754))
17179869184.9867553710937500000000000000000000000000000000000000000000000000000000000000000000000000000000000000






17179869184.98675537109375
17179869184.986755
Title: TotalPurchasers metric in Google Analytics 4 giving wrong results
Tags: python,api,google-analytics,google-analytics-api,google-analytics-4

Question: We have custom dimension define in Google Analytics Data API v1Beta for extracting data from Google Analytics GA4 account. I am trying to fetch purchaseRevenue and totalPurchasers metric with respect to date, sessionSource,  Sessionmedium, campaignName, pagePath and eventName using python. I want to know what is the Purchasers for different eventName in different campaignName. I am getting correct purchaseRevenue but more
Purchasers when i try to validate totalPurchasers data in ga4. Here is the code
import pandas as pd
import numpy as np
from google.analytics.data_v1beta import BetaAnalyticsDataClient
from google.analytics.data_v1beta.types import DateRange
from google.analytics.data_v1beta.types import Dimension
from google.analytics.data_v1beta.types import Metric
from google.analytics.data_v1beta.types import RunReportRequest

client = BetaAnalyticsDataClient()
## Format Report - run_report method
def format_report(request):
    response = client.run_report(request)
    # Row index
    row_index_names = [header.name for header in response.dimension_headers]
    row_header = []
    for i in range(len(row_index_names)):
        row_header.append([row.dimension_values[i].value for row in response.rows])
    row_index_named = pd.MultiIndex.from_arrays(np.array(row_header), names = np.array(row_index_names))
    # Row flat data
    metric_names = [header.name for header in response.metric_headers]
    data_values = []
    for i in range(len(metric_names)):
        data_values.append([row.metric_values[i].value for row in response.rows])
    output = pd.DataFrame(data = np.transpose(np.array(data_values, dtype = 'f')), 
                          index = row_index_named, columns = metric_names)
    return output
request = RunReportRequest(
        property='properties/'+property_id,
        dimensions=[            
                Dimension(name="date"),
                Dimension(name="sessionSource"),
                Dimension(name="medium"),
                Dimension(name="campaignName"),
                Dimension(name="pagePath"),
                Dimension(name="eventName"),
                ],
        metrics=[         
                Metric(name="purchaseRevenue"),
                Metric(name="totalPurchasers")
            ],
        date_ranges=[DateRange(start_date="2023-02-01", end_date="2023-02-07")],
    )

Here is the data it is showing me through api
df.totalPurchasers.sum() ```
213.0
```python
 df.purchaseRevenue.sum() ```
13710.0

but in ga4 it is showing 191 purchasers but revenue is correct.
[GA4](https://i.stack.imgur.com/SMBFY.png)

Answer: You are using right metrics.
Try increasing the limit of rows, by default it is 10K. I think after increasing rows, you will get complete data and then values will be matched.
Title: Unable to determine why eigenvalue results are different between Julia and Python for specific case
Tags: python,numpy,julia,eigenvalue

Question: I'm using Julia to do some linear algebra calculations but it gave me negative eigenvalues when I know the matrix is positive definite.
I'm fairly new to Julia so is there some reason the Julia code below would have such different behavior than the corresponding python code?  Could it be the abs function?  At this point I'm at a loss.
The Julia code is:
using LinearAlgebra

time = collect(range(0.0, 10.0, length=400)) 
H = 0.8
N = length(time)
C_N = Matrix{Float32}(undef,N,N)
for i in 1:N
    for j in 1:N 
        ti,tj = time[i], time[j]
        C_N[i,j] = 0.5*(ti^(2*H)+tj^(2*H) - abs(ti-tj)^(2*H))
    end         
end
Decomposition = eigen(C_N)
eigen_vals = Decomposition.values
has_negative = any(x -> x < 0.0, eigen_vals)
if has_negative
    @show "Has Negative eigenvalue"
else
    @show "Only positive eigenvalues"
end
has_negative

The output is:
"Has Negative eigenvalue" = "Has Negative eigenvalue"

The corresponding python code is:
import numpy as np
H = 0.8
N =400
time = np.linspace(0.0,10.0,num=N)
C_N = np.zeros((N,N))
print(time.shape)
for i in range(N):
    for j in range(N):
        ti,tj = time[i],time[j]
        C_N[i,j] = 0.5*(ti**(2*H)+tj**(2*H) - np.abs(ti-tj)**(2*H))

w, V = np.linalg.eig(C_N)
neg_mask = w < 0.0
if np.any(neg_mask):
    print("Negative eigenvalue found")
else:
    print("Only positive eigenvalues")

which outputs:
"Only positive eigenvalues"

For reference I am using Julia v"1.8.2".

Answer: It was the Float32 which led to numerical errors.  Changing to Float64 fixed it.  I'll keep the question up for posterity.
Title: pre_save django model update fith celery shared_task
Tags: python,django,django-rest-framework,celery

Question: I have Project model
class Project(models.Model):
    id = models.UUIDField(primary_key=True, unique=True, default=uuid4, editable=False)
    logo = models.ImageField(validators=[validate_image_size], blank=True, null=True, default=None)
    name = models.CharField(max_length=64)
    description = models.TextField()

    @transaction.atomic
    def save(self, *args, **kwargs):
        super().save(*args, **kwargs)

    def __str__(self):
        return self.name

And i want to compress logo field with reciever
@receiver(post_save, sender=Project)
def compress_project_logo(sender, instance, **kwargs):
    compress_image.apply_async((instance.id,))

with shared_task
@shared_task
def compress_image(project_id):
    from api.models import Project
    project = get_object_or_404(Project, id=project_id)
    compressed_image = Image.open(project.logo)
    compressed_image = compressed_image.convert("RGB")
    compressed_image = ImageOps.exif_transpose(compressed_image)

    image_io = BytesIO()
    compressed_image.save(image_io, "JPEG", quality=70)
    project.logo = InMemoryUploadedFile(image_io, "ImageField", project.logo.file.name, "image/jpeg", sys.getsizeof(image_io), None)
    project.save()

And when i'm saving Project model through django admin i take this
Traceback (most recent call last):
2023-02-28 10:27:45   

File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 450, in trace_task
  R = retval = fun(\*args, \*\*kwargs)
File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 731, in __protected_call__
  return self.run(\*args, \*\*kwargs)
File "/code/api/tasks/compress.py", line 14, in compress_image
  project = get_object_or_404(Project, id=project_id)
File "/usr/local/lib/python3.8/site-packages/rest_framework/generics.py", line 19, in get_object_or_404
  return \_get_object_or_404(queryset, \*filter_args, \*\*filter_kwargs)
File "/usr/local/lib/python3.8/site-packages/django/shortcuts.py", line 78, in get_object_or_404
  raise Http404('No %s matches the given query.' % queryset.model.\_meta.object_name)
django.http.response.Http404: No Project matches the given query.

Answer: Remove the transaction.atomic decorator I think on the save method
Title: Open 7zip with no crc in python
Tags: python,7zip,py7zr

Question: I want to open an 7zip file in python and used the py7zr library but getting following error:
CrcError: (3945015320, 1928216475, '1_Microsoft Outlook - Memoformat (3).tif')

I tried the following code:
archive= py7zr.SevenZipFile('path', mode='r',password="mypw")
archive.reset()
archive.extractall()
archive.close()

I checked with archive.test() and received None - In my understanding the crc value is missing.

Answer: The password from my client was incorrect - thank you for the help!
Title: Overwriting a method, during runtime, to raise an exception, on first pass only
Tags: python

Question: I want to rewrite, during runtime, the print method of the class below so that the first pass raises an exception, but the second time it runs normally. I cannot change the code of the Test class. Assume it's in a file I don't have access to, and I cannot mock (nor use a deepcopy, since in reality I have socket objects which are not pickable).
class Test:
    def __init__(self,a,b):
        self.a = a
        self.b = b
    def print(self):
        print(f"a-> {self.a}, and b-> {self.b}")

I tried the following (doesn't work, with infinite recursion):
test_class = Test(1,2)
test_class.num_raise = 0
copy_test = test_class
def raise_exception():
    print("Inside save_clusters")
    if test_class.num_raise ==0:
        test_class.num_raise +=1
        raise Exception("save_clusters method exception")
    elif test_class.num_raise ==1:
        return copy_test.print()

test_class.print = raise_exception

try:
    test_class.print() # should raise an exception
except:
    test_class.print() # should print as normal

Answer: How about making a wrapper around Test? You can just subclass it and override the print method with code from your raise_exception function, replacing copy_test.print() with super().print() and adding num_raise as a parameter.
Title: Can I download a large file in the background using aiohttp?
Tags: python,python-asyncio,aiohttp

Question: I'd like to download a series of large (~200MB) files, and use the time while they're downloading to do some CPU intensive processing. I'm investigating asyncio and aiohttp. My understanding is I can use them to start a large download and then do some heavy computation on the same thread while the download continues in the background.
What I am finding, however, is that the download is paused while the heavy CPU process continues, then resumes as soon as the calculation is done. I include a minimal example below. I visually monitor the process CPU and bandwidth while the script is running. It's clear the download pauses during the ~30s of computation. Am I doing something wrong? Or am I not understanding what aiohttp can do?
import asyncio
import time

import aiofiles
import aiohttp


async def download(session):
    url = 'https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-s390x.sh'  # 280 MB file
    async with session.get(url) as resp:
        async with aiofiles.open('./tmpfile', mode='wb') as f:
            print('Starting the download')
            data = await resp.read()
            print('Starting the file write')
            await f.write(data)
            print('Download completed')


async def heavy_cpu_load():
    await asyncio.sleep(5)  # make sure the download has started
    print('Starting the computation')
    for i in range(200000000):  # takes about 30 seconds on my laptop.
        i ** 0.5
    print('Finished the computation')


async def main():
    async with aiohttp.ClientSession() as session:
        timer = time.time()
        tasks = [download(session), heavy_cpu_load()]
        await asyncio.gather(*tasks)
        print(f'All tasks completed in {time.time() - timer}s')


if __name__ == '__main__':
    asyncio.run(main())

Answer: I think what happens is that the aiohttp did finish downloading the file, but in order to open it and read it1, it needs the GIL to release the lock, but the CPU load does not release the GIL until it finishes.
But, if you put await asyncio.sleep(0)2 after i ** 0.5 it will work.
await just to make sure if someone wants to take control of the GIL.

1 resp.read()
2 Common practice to release the GIL lock on purpose.
Title: How can I iterate through a doc2vec model?
Tags: python,nlp,doc2vec

Question: I have built a Doc2Vec model and am trying to get the vectors of all my testing set (176 points). The code below I can only see one vector at a time. I want to be able to do "clean_corpus[404:]" to get the entire data set but when I try that it still outputs one vector.
model.save("d2v.model")
print("Model Saved")


from gensim.models.doc2vec import Doc2Vec

model= Doc2Vec.load("d2v.model")
#to find the vector of a document which is not in training data
test_data = clean_corpus[404]
v1 = model.infer_vector(test_data)
print("V1_infer", v1)

Is there a way to easily iterate over the model to get and save all 176 vectors?

Answer: Because .infer_vector() takes a single text (list-of-words), you would want to call it multiple times in a loop if you need to infer many separate vectors for many different documents.
Another option would be to include all the documents of interest in the Doc2Vec model training data, including your test set. Then, you can simply request the learned-during-training vectors for any document, by the unique tag you supplied during training.
Whether this is an acceptable practice depends on other unstated aspects of your project goals. Doc2Vec is an unsupervised algorithm, so in some cases it can be appropriate to use all available text to improve its training. (It doesn't necessarily cause the same problems as contaminating the training of a supervised classifier with the same already-labeled examples you'll be testing it against.)
Title: Bouncing screensaver (ala DVD screensaver) sticking to edges of screen
Tags: python

Question: x = randint(50, width-60)
y = randint(50, height-60)
while True:
    x_speed = 1
    y_speed = 1
    if (x + 74 >= width) or (x <= 0):
        x_speed *= -1
    if (y + 38 >= height) or (y <= 0):
        y_speed *= -1
    x += x_speed
    y += y_speed

This code is meant to make the physics of a 'DVD screensaver'. The 74/38 numbers are the dimensions of the screensaver picture. The screensaver should bounce around the screen, colliding with the walls.
However, when I ran the program, the picture I used stuck to the wall, oscillating back and forth one pixel as it moved across the wall, eventually stopping in a corner, where it vibrated similarly.
It seems as though it keeps flipping the x_speed / y_speed variables from + to - repeatedly, which keeps it on the border, which makes it keep flipping. It should just flip once, and then bounce away.
Here are the things I tried:

Changing the dimensions of the screen: It just makes it stick to a point further away from the edge of the screen

Adding a cooldown for when it can flip the speed variables (setting a variable to a number then incrementing it back to 0 each tick before it can re-flip): It goes past the edge of the screen, jittering back a pixel each time the cooldown ends.

Changing the dimensions of the picture (not visually, but in the physics): This only made it stick slightly past the edge of the screen.


I cannot think of anything else to try. I have looked for alternative code, but none of it works with my program/it isn't in Python. Can anyone see any problems with the code?

Answer: Because of the while loop, the variables x_speed and y_speed are being instanced over and over again in each loop and don't change.
Instead, put these variables outside the while loop so they don't get instanced again.
Title: How to install Kivy with dependencies in Windows 10 using pip?
Tags: python,android,windows,kivy

Question: error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [10 lines of output]
Collecting setuptools
Using cached setuptools-67.4.0-py3-none-any.whl (1.1 MB)
Collecting wheel
Using cached wheel-0.38.4-py3-none-any.whl (36 kB)
Collecting cython!=0.27,!=0.27.2,<=0.29.28,>=0.24
Using cached Cython-0.29.28-py2.py3-none-any.whl (983 kB)
Collecting kivy_deps.gstreamer_dev~=0.3.3
Using cached kivy_deps.gstreamer_dev-0.3.3-cp311-cp311-win_amd64.whl (3.9 MB)
ERROR: Could not find a version that satisfies the requirement kivy_deps.sdl2_dev~=0.4.5 (from versions: 0.5.1)
ERROR: No matching distribution found for kivy_deps.sdl2_dev~=0.4.5
[end of output]
I was trying to install kivy with "pip install kivy[full]" but isted of installing kivy suprocess error occored then I tryed installing subprocess with "pip install subprocess.run" it was installed sucessfully but again the same error is occruing

Answer: To fix this error, try to install Kivy using the pre-built wheel from the Kivy website. Download the wheel and install it using the pip command:
pip install <path-to-wheel-file>
Title: PySpark in Databricks error with table conversion to pandas
Tags: python,pandas,apache-spark,pyspark,databricks

Question: I'm using Databricks and want to convert my PySpark DataFrame to a pandas one using the df.toPandas() command.
However, I keep getting this error:
/databricks/spark/python/pyspark/sql/pandas/conversion.py:145: UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
  'DataFrame' object has no attribute 'dtype'
  warnings.warn(msg)
AttributeError: 'DataFrame' object has no attribute 'dtype'

I tried different things, including:
spark.conf.set("spark.sql.execution.arrow.enabled", "false")

But nothing worked so far (I also checked some of the other posts that have this issue, but none helped).
UPDATE: result of df.printSchema():
flight_id: string (nullable = true)
 |-- flight_direction: string (nullable = true)
 |-- service_type: string (nullable = true)
 |-- flight_designator: string (nullable = true)
 |-- flight_number: string (nullable = true)
 |-- callsign: string (nullable = true)
 |-- scheduled_datetime: timestamp (nullable = true)
 |-- connecting_flight_designator: string (nullable = true)
 |-- airport_iata_codes: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- airline_name: string (nullable = true)
 |-- airport_names: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- country_number: long (nullable = true)
 |-- eu_category: string (nullable = true)
 |-- safe_town_indicator: boolean (nullable = true)
 |-- sibt: timestamp (nullable = true)
 |-- aibt: timestamp (nullable = true)
 |-- sobt: timestamp (nullable = true)
 |-- aibt: timestamp (nullable = true)
 |-- tsat: timestamp (nullable = true)
 |-- aircraft_name: string (nullable = true)
 |-- aircraft_registration: string (nullable = true)
 |-- ramp: string (nullable = true)
 |-- ramp_previous: string (nullable = true)
 |-- seats: long (nullable = true)
 |-- actual_total_pax: integer (nullable = true)
 |-- handler_apron: string (nullable = true)
 |-- occupancy_rate: double (nullable = false)

Answer: There was a problem in the data filtering. There were duplicate columns. If anyone in the future has a similar issue, please check this.
Title: How do I solve "error: externally-managed-environment" everytime I use pip3?
Tags: python,error-handling,pip

Question: error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I use apt upgrade and update.

Answer: # rm /usr/lib/python3.11/EXTERNALLY-MANAGED
Title: How do I solve "error: externally-managed-environment" everytime I use pip3?
Tags: python,error-handling,pip

Question: error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I use apt upgrade and update.

Answer: that issue from pip just run the command and it will downgrade it.
=> pip install pip==22.3.1 --break-system-packages
surely that will help.
Title: How do I solve "error: externally-managed-environment" everytime I use pip3?
Tags: python,error-handling,pip

Question: error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I use apt upgrade and update.

Answer: To install the package XXX, instead of
pip install XXX
try:
sudo apt install python3-XXX
Title: How do I solve "error: externally-managed-environment" everytime I use pip3?
Tags: python,error-handling,pip

Question: error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I use apt upgrade and update.

Answer: Happens to me after I changed the name of the folder containing my virtual environment
If you want to rename a virtual environment, changing the name of the folder doesn't work, what I did was re-creating it