Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Using tf.data.Dataset prefetch makes model performance overfit?
Tags: python,tensorflow,machine-learning,deep-learning,prefetch

Question: I'm trying to train a simple LRCN model with some sequential image dataset in Tensorflow 2.5.0.
Training performance was fine like increasing to 0.9x training & validation accuracy both in first 5 epochs and train & validation loss kept decreasing during the training.
Then, I've tried to optimize data pipeline with using prefetch().
The dataset I use is sequential images (.png) that titles and information are written in .csv file. So I made the data generator like below :
def setData(data):
X, y = [], []

name = data.loc['fileName'].values.tolist()[0]
info1 = data.loc['info1'].values.tolist()[0]
info2 = data.loc['info2'].values.tolist()[0]
info3 = data.loc['info3'].values.tolist()[0]

if os.path.isfile(filepath + name) == False:
    print('No file for img')
    return

try:
    img = np.load(filepath + fName)
except:
    print(name)  

if info1 in info_list:  
    X.append(img)

    if info2 == 'True':
        y.append(0)

    else:
        y.append(1)

X = np.array(X)
X = np.reshape(X, (3, 128, 128, 1)).astype(np.float64)
y = np_utils.to_categorical(y, num_classes = 2)
y = np.reshape(y, (2)).astype(np.float64)

return X, y

And I added the data generator load function like this :
def generatedata(i):
    i = i.numpy()
    X_batch, y_batch = setData(pd.DataFrame(traindata.iloc[i]))

    return X_batch, y_batch

Finally, I prefetched dataset using map
z = list(range(len(traindata[])))
trainDataset = tf.data.Dataset.from_generator(lambda: z, tf.uint8)
trainDataset = trainDataset.map(lambda i: tf.py_function(func = generatedata,
                                                         inp = [i],
                                                         Tout = [tf.float32, tf.float32]),
                                                         num_parallel_calls = tf.data.AUTOTUNE)

After I applied these steps, training accuracy goes 0.9x in first epoch, 1.0 in the first 3-5 epochs and validation accuracy stays at around 0.6x and validation loss kept growing over x.x.
I believe that the prefetch only changes the data pipeline that do not affect to the model performance so I'm not sure what caused this overfitting(maybe?)-like results.
I followed every step of the prefetch step that were denoted at the Tensorflow documentation. Though, since I'm not very familiar with tensorflow, there might be some mistakes.
Is there any line that I missed? Any opinion would be really greatfull.
Thanks in advance.

Answer: It turns out that the py_function() makes tf.graph stacked over previous results that leads the model to overfitting.
I've modified the prefetch function to get the generator function and works as it should be.
Though I checked the tensorflow documents, I haven't fully alarmed of this situation but found this in tensorflow github page.
To those who have same problem as me, try to review the library module function carefully.
Title: Handling CSR data file
Tags: python,numpy,sparse-matrix

Question: I have sparse matrix data stored in npy format. I am reading the file and storing it in CSR format. And the below code works perfectly:
def load_adjacency_matrix_csr(folder: str,
                              filename: str,
                              suffix: str = "npy",
                              row_idx: int = 1,
                              num_rows: int = None,
                              dtype: np.dtype = np.float32) -> sparse.csr_matrix:
    coo_indices = np.load(os.path.join(folder, f"{filename}.{suffix}"))
    rows = coo_indices[row_idx]
    cols = coo_indices[1 - row_idx]
    data = np.ones(len(rows), dtype=dtype)

    num_rows = num_rows or rows.max() + 1
    if num_rows < rows.max() + 1:
        raise ValueError("The number of rows in the file is larger than the specified number of rows.")
    csr_mat = sparse.csr_matrix((data, (rows, cols)), shape=(num_rows, num_rows), dtype=dtype)
    return csr_mat

Now I read the same file, write it to a new file (_temp) and then I read. But now when I read from this new file (_temp), I get an error Detected 1 oom-kill even(s). This out of memory happens when creating the CSR matrix. But this is the exact same number of rows/columns and data as when I was able to successfully execute when I directly from the file. I am unable to figure out what changed when writing to a new file (the rows/columns and data has not changed).
def save_coo_matrix(npx: int,
                    folder: str,
                    filename: str,
                    suffix: str = "npy",
                    row_idx: int = 1,
                    dtype: np.dtype = np.float32) -> None:

  file_path = os.path.join(folder, f"{filename}.{suffix}")
  coo_indices = np.load(file_path)
  rows = coo_indices[row_idx]
  cols = coo_indices[1 - row_idx]
  filename += "_temp"
  file_path = os.path.join(folder, f"{filename}.npz")
  np.savez(file_path, rows=rows, cols=cols)

def load_adjacency_matrix_csr(npx: int,
                              folder: str,
                              filename: str,
                              suffix: str = "npy",
                              row_idx: int = 1,
                              num_rows: int = None,
                              dtype: np.dtype = np.float32) -> sparse.csr_matrix:
    file_path = os.path.join(folder, f"{filename}_padded.npz")
    with np.load(file_path) as data:
        rows = data["rows"]
        cols = data["cols"]
        num_rows = max(rows) + 1
        num_cols = max(cols) + 1
        values = np.ones(len(rows))
        # fails here when creating the csr matrix (the num_rows is the same value as before)
        csr_mat = sparse.csr_matrix((values, (rows, cols)), shape=(num_rows, num_cols))

    return csr_mat

Answer: There seem to be a mismatch in the file extensions you use when writing and reading the data. when you call load_adjacency_matrix_csr, make sure to set the suffix parameter to csr_matrix = load_adjacency_matrix_csr(npx, folder, filename, suffix="npz", ...) also the npx is not used within the function. You can remove this parameter from the function definition.
Title: azure information protection remove label in linux
Tags: python,linux,azure

Question: I am in a situation that requires (decrypting/removeLabel) in Linux eg. Debian or rasbian ?
I have looked into azure products and they clearly stated that there is no "official" support for Linux..... but my question here is .... is there any workaround to achieve this in Linux?
One silly way I could think of is to get this protected file in a windows pc, decrypt it and send it to the Linux PC over scp.... but this requires me to have a sort of a bypass PC to run just for this ...seems pretty silly ... any ideas/suggestion?
Thanks in advance

Answer: Azure Information Protection dont have an official client for linux, one solution would be to use a windows virtual machine on your linux system and install the AIP client in it, VirtualBox or Vmware are good choices here. You can also use the AIP PowerShell module to automate the removal of labels
Title: Django and Time Zone for regions that no longer observe the Daylight Save Time (DST)
Tags: python,django,timezone,dst

Question: I have a Django project, time zone is enabled for multiple purposes, my TIME_ZONE = 'America/Mexico_City', however, since this year 2023, in this time zone it's no longer observed the DST, I use the localtime to get the right date/time in some cases, but it detects the DST
>>> localtime(timezone.now())
datetime.datetime(2023, 4, 3, 10, 14, 49, 782365, tzinfo=<DstTzInfo 'America/Mexico_City' CDT-1 day, 19:00:00 DST>)
>>> timezone.now()
datetime.datetime(2023, 4, 3, 15, 14, 54, 953013, tzinfo=<UTC>)
>>> datetime.now()
datetime.datetime(2023, 4, 3, 9, 15, 7, 628038)

datetime.now() has the correct date/time, of course I can change the localtime for datetime.now(), but there are a lot of them, I want to understand how I can "update" or "sync" my Django project so it take the correct DST when some region change it for both observe or not longer observe it.

Answer: UPDATE: Solved
Thanks to @FObersteiner for the reference, my pytz version was old. Simply upgrade it with pip install --upgrade pytz, that will solve the issue.
Title: TypeError: delete_dog() missing 1 required positional argument: 'Dog_id' How do I get around this?
Tags: python,sql,flask

Question: I have written this code and have gotten this TypeError and don't understand the error. What am I doing wrong.
    
@app.route("/deletedog", methods=["GET", "POST"])
def delete_dog(Dog_id):
    p=dog.query.filter_by(Dog_id=Dog_id).first()
    if request.method == 'GET':
        db.session.delete(p)
        db.session.commit()
        return redirect(url_for('index'))
    return render_template("deletedog.html")

`
Im kinda of scratching my head i didn't think it needs a positional

Answer: I assume you are making a Flask app, kindly, next time include those details in your question. To add parameters to a flask app you need to include in the route URL the request, so changing the route url to something like /deletedog/<dog_id> should make this to work.
Title: Python wont run inside CMD, PS, or VS Code
Tags: python,powershell,visual-studio-code,terminal,command-prompt

Question: I am a learner, so please forgive me if this is common knowledge
I fixed my PATH files for python, I made the only PATH and sole instance of Python C:\Python
This fixed the issue where VS code could not find a python interpreter.
Now that I have done this, when when i try to run a python command, this happens:
PS C:\Users\sajja\Documents\git\cs50w_repo\cs50w_lecturepractice\lecture2-python> python name.py
but it never prompts me for the input, which is the whole point of this file.
when i check the python install, vscode does recognize it, just wont run it

PS C:\Users\sajja\Documents\git\cs50w_repo\cs50w_lecturepractice\lecture2-python> python
Python 3.11.2 (tags/v3.11.2:878ead1, Feb  7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

If i switch to the output tab and do it from there instead, it runs the code without error, but never prompts me for the input, and therefore never prints the output

[Running] python -u "c:\Users\sajja\Documents\git\cs50w_repo\cs50w_lecturepractice\lecture2-python\name.py"

[Done] exited with code=0 in 0.057 seconds
I was expecting for this code:
name = input("Name: ")
print(name)

to prompt me to enter an input, and then print said input in the output or terminal

Answer: There are many similar questions, but I could'nt find an exact one.
Here is the solution I found:
Instead of running the file in VSCode, I selected Run and Debug, which opened a different terminal (maybe?), this terminal accepts input, and therefore printed the correct output.
Here it is:

PS C:\Users\sajja\Documents\git\cs50w_repo\cs50w_lecturepractice> & C:/Python/python.exe c:/Users/sajja/Documents/git/cs50w_repo/cs50w_lecturepractice/lecture2-python/name.py
Name: sajjad
sajjad

The output window still does not do what it is supposed to, but the Terminal window is behaving the way it should
Title: Databricks: Issue while creating spark data frame from pandas
Tags: python,pandas,apache-spark,databricks,iteritems

Question: I have a pandas data frame which I want to convert into spark data frame. Usually, I use the below code to create spark data frame from pandas but all of sudden I started to get the below error, I am aware that pandas has removed iteritems() but my current pandas version is 2.0.0 and also I tried to install lesser version and tried to created spark df but I still get the same error. The error invokes inside the spark function. What is the solution for this? which pandas version should I install in order to create spark df. I also tried to change the runtime of cluster databricks and tried re running but I still get the same error.
import pandas as pd
spark.createDataFrame(pd.DataFrame({'i':[1,2,3],'j':[1,2,3]}))

error:-
UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below:
  'DataFrame' object has no attribute 'iteritems'
Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true.
  warn(msg)
AttributeError: 'DataFrame' object has no attribute 'iteritems'

Answer: It's related to the Databricks Runtime (DBR) version used - the Spark versions in up to DBR 12.2 rely on .iteritems function to construct a Spark DataFrame from Pandas DataFrame.  This issue was fixed in the Spark 3.4 that is available as DBR 13.x.
If you can't upgrade to DBR 13.x, then you need to downgrade the Pandas to latest 1.x version (1.5.3 right now) by using %pip install -U pandas==1.5.3 command in your notebook. Although it's just better to use Pandas version shipped with your DBR - it was tested for compatibility with other packages in DBR.
Title: Databricks: Issue while creating spark data frame from pandas
Tags: python,pandas,apache-spark,databricks,iteritems

Question: I have a pandas data frame which I want to convert into spark data frame. Usually, I use the below code to create spark data frame from pandas but all of sudden I started to get the below error, I am aware that pandas has removed iteritems() but my current pandas version is 2.0.0 and also I tried to install lesser version and tried to created spark df but I still get the same error. The error invokes inside the spark function. What is the solution for this? which pandas version should I install in order to create spark df. I also tried to change the runtime of cluster databricks and tried re running but I still get the same error.
import pandas as pd
spark.createDataFrame(pd.DataFrame({'i':[1,2,3],'j':[1,2,3]}))

error:-
UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below:
  'DataFrame' object has no attribute 'iteritems'
Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true.
  warn(msg)
AttributeError: 'DataFrame' object has no attribute 'iteritems'

Answer: This issue is occurred due to pandas version <= 2.0. In Pandas 2.0, .iteritems function is removed.
There are two solutions for this issue.

Down grade the pandas version < 2. For example,

pip install -U pandas==1.5.3

Use the latest Spark version i.e 3.4
Title: Pytest AttributeError: 'TestMyTest' object has no attribute
Tags: python,pytest

Question: I have multiple tests written in the following format. When the test is run, why do some tests fail with exception AttributeError: 'TestMyTest' object has no attribute 'var1'?
var1 is defined as class variable in test_init(). Still the exception is thrown in test_1().
@pytest.fixture(scope="module")
def do_setup(run_apps):
    """
    Do setup
    """
    # Do something
    yield some_data


class TestMyTest(BaseTestClass):
    ids = dict()
    task_id = list()
    output_json = None

    def test_init(self, do_setup):
        TestMyTest.var1 = Feature1()

    def test_1(self):
        self._logger.info("self.var1: {}".format(self.var1))

Answer: var1 was not defined as test_init() failed.
Title: One loop with all lists in one instead of two loops with separated lists
Tags: python,algorithm

Question: I have an algorithmic question. Sorry if it seems to you silly.
Consider I have two lists:
list_a = ["Red", "Blue", "Black"]
list_b = ["Samsung", "Apple"]

So the question is, if I iterate through the lists above with two for loop is faster or merging them and using just one?

Answer: Imho it depends on what you're trying to accomplish with the iteration.
If you need to perform an operation on each combination of elements from list_a and list_b, such as creating a Cartesian product or a nested loop, then using two for loops is necessary. In this case, merging the lists would not be useful and may even cause errors in your code.
However, if you just need to iterate over all the elements in both lists sequentially, then merging them into one list and using a single for loop would be faster and more efficient.
Title: Reading modbus registers
Tags: python,modbus,modbus-tcp,pymodbus

Question: I have python script which reads registers from energy meter and saves values to database.
Script was working fine until today when i tried to run it and i got error:
AttributeError: 'ModbusIOException' object has no attribute 'registers'

I can ping device normally...
This is my code (half of it) - even simple print of value doesn't work anymore
from pymodbus.client import ModbusTcpClient

IP = "192.168.X.X"
client = ModbusTcpClient(IP)

reg = client.read_holding_registers(23322, 2)
calc = round((reg.registers[0] * pow(2, 16) + reg.registers[1]) * 0.01 / 1000, 2)

print(calc)

What could be the problem?

Answer: Your device is obviously returning an exception instead of the values, have a look at the type of the exception (by looking into your reg variable) to get a clue on how to fix it
Title: Successfully installed pip, but error displayed when viewing pip version
Tags: python,pip,scapy

Question: The command line displays successful installation of pip, but when querying the pip version, it displays Script file 'D:\anaconda\Scripts\pip-script.py' is not present.I don't know how to resolve it. Help!!!
I tried many methods, but it seems like I can't use pip. I wanted to call scapy, but I can't use pip anymore.Please help me!

Answer: Did you check the PATH variable in your windows? Check that the pip executable is in your PATH variable. To do this, open a command prompt and type echo %PATH%. Look for the directory where pip is installed (likely something like C:\Python3\Scripts).
Now if pip is in your PATH, try running it directly from the command prompt by typing pip and pressing Enter. But If this still gives an error, I think you should try specifying the full path to pip.
Or you should re install it
Title: Python server is killed when started using docker-compose exec
Tags: python,ubuntu,docker-compose

Question: I start docker container and want to start simple python server in it.
I can't use ENTRYPOINT/CMD of Dockerfile as it is already used for other things.
I am trying to do it the following way:
docker-compose exec <service_name> /bin/bash -c "./server.py &"

But as soon as docker-compose exec ends server stops.
Yet if I run application written on C++ in the same way it continues to run:
docker-compose exec <service_name> /bin/bash -c "./app &"

I inspected this using htop and found out that both server.py and app start as child processes but after end of docker-compose exec app reattaches and server.py stops.

How can I make my python server continue to run starting it through `docke-compose exec?

Answer: Using nohup like /bin/bash -c "nohup ./server.py & as proposed in commentaries did help.
Title: FastAPI - switching off a switch some time later
Tags: python,fastapi,pydantic,sqlmodel

Question: Let's say I have this sqlmodel.SQLModel which has a table name Project.
My intention is to feature a record of a Project for a definite period of time, e.g. 3 days, i.e. setting it's field featured to be True, and automatically set its field featured to be False thereafter.
class Project(SQLModel):
    featured: bool = False

How can I achieve this behavior using FastAPI? Is it via background tasks or what?

Answer: Consider using something like "featured until" instead. The field will contain a date stating the last date the project should be featured. Your feature page will then look for any projects with a "features until" date greater than today.
This way you don't need any background process so your system will be simpler and easier to maintain. It also leaves you with feature history in the database which may be useful in the future.
Title: snscrape error when attempting to scrape from mastodon
Tags: python,mastodon

Question: I'm not sure what I'm doing wrong. I'm trying to test out snscrape for scraping posts from Mastodon, using @pomological@botsin.space as a test case. (A bot that posts public domain fruit illustrations.) I think I'm following the correct format but it keeps giving me this error. Can anyone help? It gives me a similar error when I try either the "mastodon-profile" or "mastodon-toots" scrapers. I have used snscrape to scrape tweets from twitter many times and haven't had this problem.
The command I'm using is snscrape mastodon-profile @pomological@botsin.space.
C:\Windows\system32>snscrape mastodon-profile @pomological@botsin.space
2023-04-04 11:49:23.086 CRITICAL snscrape._cli Dumped stack and locals to C:\Users\[me]\AppData\Local\Temp\snscrape_loca
ls_3xbv51be
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\[me]\AppData\Local\Programs\Python\Python311\Scripts\snscrape.exe\__main__.py", line 7, in <module>
  File "C:\Users\[me]\AppData\Local\Programs\Python\Python311\Lib\site-packages\snscrape\_cli.py", line 320, in main
    for i, item in enumerate(scraper.get_items(), start = 1):
  File "C:\Users\[me]\AppData\Local\Programs\Python\Python311\Lib\site-packages\snscrape\modules\mastodon.py", line 279, in get_items
    yield from self._entries_to_items(soup.find('div', class_ = 'activity-stream').find_all('div', class_ = 'entry'), r.url)
AttributeError: 'NoneType' object has no attribute 'find_all'

I've done some minimal python programming, but I'm pretty out of my depth here. Any assistance would be appreciated. I may have missed something obvious!

Answer: Feels like either the Mastodon profile page of this account does not have the expected HTML structure that snscrape is looking for. Either that or there might be server-side issues on Mastodon's end that are preventing snscrape from retrieving the data.
Title: Debugger in Pycharm doesn't work properly
Tags: python,ide

Question: I'm using Pycharm and want to debug my project but when I go and hit the debug key I get the following message
Python Debugger Extension Available
Cython extension speeds up Python debugging
When I click on install I get the following error message
Copile Cython Extension Error:

Non-zero exit code (1): Traceback (most recent call last): File "/opt/pycharm-professional/plugins/python/helpers/pydev/setup_cython.py", line 242, in   main() File "/opt/pycharm-professional/plugins/python/helpers/pydev/setup_cython.py", line 233, in main with FrameEvalModuleBuildContext(): File "/opt/pycharm-professional/plugins/python/helpers/pydev/setup_cython.py", line 187, in __enter__ shutil.copy(compatible_c, self._c_file) File "/home/lz/.pyenv/versions/3.8.16/lib/python3.8/shutil.py", line 418, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/home/lz/.pyenv/versions/3.8.16/lib/python3.8/shutil.py", line 264, in copyfile with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied: '/opt/pycharm-professional/plugins/python/helpers/pydev/_pydevd_frame_eval/pydevd_frame_evaluator.c' 

(Side notes: I'm using arch and also utilize pyenv for python version management)
I tried to install cython both via Arch's package manager and via pip in my project but to no success

Answer: Run
sudo chown <user> <path_to_dir>/_pydevd_frame_eval/
Open up pycharm again as a normal user, click on install and everything should work as intended
Title: “AttributeError: Can't get attribute 'Document' on <module 'langchain.schema' on Google Colab
Tags: python,google-colaboratory,pickle,langchain

Question: When I try to load a pickle file onto Google Colab like this:
with open("file.pkl", "rb") as f:
        vectorstore = pickle.load(f)

I either get:
AttributeError: Can't get attribute 'Document' on <module 'langchain.schema' from '/usr/local/lib/python3.9/dist-packages/langchain/schema.py'>

or
ImportError: cannot import name 'Document' from 'langchain.schema' (/usr/local/lib/python3.9/dist-packages/langchain/schema.py)

Reinstalling Langchain and other workarounds did not fix the issue. The .pkl file is already in my "files" tab, so it's not an issue with importation

Answer: I had the same error and recreating the pkl file solved it. If you updated your environment after creating the pkl file it's likely a problem with different versions of a module (I couldn't spot which in my case)
Title: How can I debug IPython display methods?
Tags: python,jupyter-notebook,ipython,jupyter-lab

Question: I'm writing _repr_latex_, _repr_html_, __repr__ methods.
The problem that I'm facing is that I want to debug the call chain, because I bumped in to a situation like this.
class A:
    def _get_text(self):
        return self.__class__.__name__

    def _repr_html_(self):
        text = self._get_text()
        return f"<b>{text}</b>"

class B(A):
    def __repr__(self):
        return f"<{self.__class__.__name__} >"

class C(B):
    def _get_text(self):
        return "I'm C, an equivalent data structure with different meaning."


And whenever I try to display by stating an object c = C at the end of a code block, it displays the __repr__ not the _repr_html_, corresponding to the class C.
If anyone knows what is a common debugging process for this situation?

Answer: After examining all the methods, mentioned, I realized that there was an exception occurring in one of the methods, forcing a fallback to __repr__.
The conclusion is to make sure that the _repr_{type}_ doesn't raise exceptions.
Title: self() as function within class, what does it do?
Tags: python,pytorch

Question: Sorry for the poor title but I'm unsure how better to describe the question.
So I recently watched Andrej Kaparthy's build GPT video which is awesome and now trying to reconstruct the code myself I notices that he uses self() as a function and was curious why and what exactly it does.
The code is here and I'm curious in particular about the generate function:
class BigramLanguageModel(nn.Module):

    def __init__(self, vocab_size):
        super().__init__()
        # each token directly reads off the logits for the next token from a lookup table
        self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)

    def forward(self, idx, targets=None):

        # idx and targets are both (B,T) tensor of integers
        logits = self.token_embedding_table(idx) # (B,T,C)

        if targets is None:
            loss = None
        else:
            B, T, C = logits.shape
            logits = logits.view(B*T, C)
            targets = targets.view(B*T)
            loss = F.cross_entropy(logits, targets)

        return logits, loss

    def generate(self, idx, max_new_tokens):
        # idx is (B, T) array of indices in the current context
        for _ in range(max_new_tokens):
            # get the predictions
            logits, loss = self(idx)
            # focus only on the last time step
            logits = logits[:, -1, :] # becomes (B, C)
            # apply softmax to get probabilities
            probs = F.softmax(logits, dim=-1) # (B, C)
            # sample from the distribution
            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)
            # append sampled index to the running sequence
            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)
        return idx


So to me it seems that he is calling the forward function defined within the class through using the self(). Is that correct? And if so why would he not use forward(idx) instead. Thank you for you help!

Answer: Meh, this is pytorch. Remember that you can use the model like this: model(x) to do the model.forward(x). So inside of the model class self(x) will be the basically the same as doing self.forward(x).
Title: Poetry install ChefBuildError
Tags: python-3.x,python-poetry

Question: Poetry install fails with ChefBuildError: Backend operation failed: HookMissing('build_editable')
My poetry version is 1.4.2

Answer: This worked for me
I believe this is caused by a change to how the build-backend is defined in the pyproject.toml between poetry ^1.3 and poetry ^1.4. Assuming you have poetry ^1.4 installed you have two options:

In your pyproject.toml change build-backend = "poetry.masonry.api" to build-backend = "poetry.core.masonry.api"

If, like me you have other code the assumes poetry ^1.3 then simply downgrade you poetry version poetry self update 1.3.2



If you go with option 2 you may get a bunch of RuntimeError hash for xxx errors. If that's the case you will also need to rm -r ~/.cache/pypoetry/artifacts and rm -r ~/.cache/pypoetry/cache.
Title: Poetry install hash for package.whl not found in known hashes
Tags: python-3.x,python-poetry

Question: When running poetry install I get the following error
Hash for some-package (a.b.c) from archive some-package-a.b.c....whl not found in known hashes
poetry version ^1.3

Answer: This worked for me.
rm -r ~/.cache/pypoetry/cache and failing that maybe try rm -r ~/.cache/pypoetry/artifacts
Then ruunning poetry install again.
Title: Increasing programming efficiency to O(n) from O(n^2)
Tags: python,algorithm,time-complexity

Question: Let's say you have an array of numbers and a queries array:
array = [1,3,5,6,9]
queries = [5,9,3]

We need to find the number of operations required i.e. add 1 or subtract 1 from the array to bring all numbers of the array to match the element from the queries array.
For example:

For query 5, number of operations required to bring all elements of array to 5 is abs(1-5) + abs(3-5) + abs(5-5) + abs(6-5) + abs(9-5) = 4+2+0+1+4 = 11.

For query 9, it would be 21.

For query 3 it would be 13.


Is there a way to do this with time complexity better than O(len(array) x len(queries)) ?
My function to do this in python is as follows
def find_min_costs(array,queries):
    for query in queries:
        cost = 0
        for arr in array:
            cost+= abs(arr-query)
        print(cost,end = ' ')

but the time complexity of my code is O(len(array) x len(queries))
I would like to do this with a better time complexity such as O(len(array))

Answer: Sort the input array (if it is not sorted already).
Now, for every query q, find the position pos where the query number would be placed in the array (use binary search).
Next, observe that, for all numbers to the left, array[1], ..., array[pos], their impact would be q - array[i]. The total is pos * q - sum_of_array_from_1_to_pos.
Similarly, for all numbers to the right, array[pos+1], ..., array[n], their impact would be array[i] - q. The total is sum_of_array_from_pos+1_to_n - (n-pos) * q.
If you compute prefix sums of the sorted array in advance, the above quantities can be found in O(1).
In total, the preparation step is sorting the array in O(n log n), where n is the length of the array.
Then computing prefix sums in O(n).
For each query, the work can be done in O(log n) for binary search and then O(1) for computing the answer. The total is O(m log n), where m is the number of queries and n is the length of the array.

How to use prefix sums: if we have prefix sums p[i] = array[1] + array[2] + ... + array[i], then a quantity like sum_of_array_from_x_to_y is just p[y] - p[x-1].
How to compute prefix sums: p[0] = 0, p[i] = p[i - 1] + array[i].
Title: Use while loop to check for the condition
Tags: python,python-3.x,while-loop

Question: I was trying to make this program in which every time when the users enter a number, if the number is in the range between 0 and 2, it will return the number the user has inputed. Otherwise, it will keep printing out the statement "Enter selection: " to the console until the user has entered a value that is in the range. My problem is it seems like my while loop is not working so that every time no matter what value I enter (even it's outside the range 0, 2) the value still returns.
Below is my existing code:
def get_user_option_input():
    number = input('Please enter a number: ')
    num = int(number)
    while not num > 0 and num < 2:
        print('Enter selection: ')
    return num
 
number = get_user_option_input()
print(number)

Hope someone can help me figure this out. Much appreciated!

Answer: If the first number the user inputs is outside the range, then it will run print('Enter selection: ') forever, because num is never changed inside the loop. To fix this you need to change print('Enter selection: ') to num = int(input('Enter selection: ')). Also, change while not num > 0 and num < 2:to while not (num > 0 and num < 2):. Otherwise, not only negates num > 0. By adding a parentheses, not negates both num > 0 and num < 2.
Title: Error while importing gensim library value error
Tags: python,nlp,gensim,word2vec

Question: while i am importing the library gensim
import gensim
it throws this error
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_14076/1427573704.py in <module>
----> 1 import gensim

~\AppData\Local\Programs\Python\Python310\lib\site-packages\gensim\__init__.py in <module>
      9 import logging
     10 
---> 11 from gensim import parsing, corpora, matutils, interfaces, models, similarities, utils  # noqa:F401
     12 
     13 

~\AppData\Local\Programs\Python\Python310\lib\site-packages\gensim\corpora\__init__.py in <module>
      4 
      5 # bring corpus classes directly into package namespace, to save some typing
----> 6 from .indexedcorpus import IndexedCorpus  # noqa:F401 must appear before the other classes
      7 
      8 from .mmcorpus import MmCorpus  # noqa:F401

~\AppData\Local\Programs\Python\Python310\lib\site-packages\gensim\corpora\indexedcorpus.py in <module>
     12 import numpy
     13 
---> 14 from gensim import interfaces, utils
     15 
     16 logger = logging.getLogger(__name__)

~\AppData\Local\Programs\Python\Python310\lib\site-packages\gensim\interfaces.py in <module>
     17 import logging
     18 
---> 19 from gensim import utils, matutils
     20 
     21 

~\AppData\Local\Programs\Python\Python310\lib\site-packages\gensim\matutils.py in <module>
   1028 try:
   1029     # try to load fast, cythonized code if possible
-> 1030     from gensim._matutils import logsumexp, mean_absolute_difference, dirichlet_expectation
   1031 
   1032 except ImportError:

~\AppData\Local\Programs\Python\Python310\lib\site-packages\gensim\_matutils.pyx in init gensim._matutils()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject`

What is the solution of this error?

Answer: This likely means you've installed mismatched versions of libraries that closely depend on each other, like Gensim and Numpy.
In such cases, as ~zanga suggests, starting from a fresh Python environment with no optional libraries yet installed is likely to help. As you install just what you need, you'll get the latest/compatible libraries.
In general it is best to use independent Python virtual environments for your own separate coding project, as supported by the Python 3 venv functionality, or also well-implemented by the conda package/environment management too.
Then, you don't change the shared system Python environment with mixed mismatches of libraries that can risk:

library choices you made in an earlier project negatively impacting later projects
new libraries in later project inadvertently breaking earlier projects, which have their preferred versions changed out-from-under-them
being unsure what libraries/versions are needed for a project when you move it elsewhere or share it with others (because sufficient libraries just happened to already be there without proper explicit declaration)
Title: Speedup Sklearn model fitting
Tags: python,performance,scikit-learn,jupyter-notebook,thread-priority

Question: I am profiling Sklearn model:
clf = GridSearchCV(..., n_jobs=-1)   
%time clf.fit(X_train, y_train)
...

CPU times: user 2min 35s, sys: 3.07 s, total: 2min 38s
Wall time: 8min 40s

Wall Time is significantly larger than CPU total time.

Does it mean, that Sklearn is not fully utilizing CPU resources? I
haven't any programs on my PC started explicitly, except Jupyter
Notebook.

How do I can increase CPU priority for all processes, that Sklearn
have started?


OS: Kubuntu 22.04

Answer: Much higher wall time than CPU time usually indicates an I/O bottleneck, but that should not happen training a scikit-learn model on data you have in memory. The next thing I would try is setting n_jobs to the number of physical CPU cores.
Title: Anaconda Python 3.9 (Windows): Why can't CVXPY find qdldl.dll?
Tags: python,conda,cvxpy

Question: I'm running Anaconda Python 3.9 on Windows 10:
Python 3.9.16 (main, Mar  8 2023, 10:39:24) [MSC v.1916 64 bit (AMD64)] on win32

I installed CVXPY in an Anaconda environment using the command:
conda install -c conda-forge cvxpy

I then attempted to run the test suite via
(qutip-env) C:\Users\ray>pytest --pyargs cvxpy.tests

and I obtain
================================================= test session starts =================================================
platform win32 -- Python 3.9.16, pytest-7.1.2, pluggy-1.0.0
rootdir: C:\Users\ray
plugins: rerunfailures-10.1
collecting ... (CVXPY) Apr 05 12:27:19 PM: Encountered unexpected exception importing solver OSQP:
ImportError('DLL load failed while importing qdldl: The specified module could not be found.')
collecting 0 items                                                                                                     (CVXPY) Apr 05 12:27:20 PM: Encountered unexpected exception importing solver OSQP:
ImportError('DLL load failed while importing qdldl: The specified module could not be found.')
collecting 1189 items                                                                                                  (CVXPY) Apr 05 12:27:20 PM: Encountered unexpected exception importing solver OSQP:
ImportError('DLL load failed while importing qdldl: The specified module could not be found.')
collected 1201 items

I assumed that the problem was that the directory containing the DLL qdldl.dll wasn't on my PATH, so I updated my system environment variable to include it:
C:\Users\ray\anaconda3\envs\qutip-env\Library\bin

Closing and restarting the Anaconda prompt, I use
(qutip-env) C:\Users\ray>echo %PATH%

and I see this directory. Python sees it as well if I enter:
(qutip-env) C:\Users\ray>python
Python 3.9.16 (main, Mar  8 2023, 10:39:24) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> import os
>>> print(os.environ['PATH'])

But I see the same "failure to load" error when I test CVXPY again. Is there another way to tell Python/CVXPY where to look for this DLL?
(I actually first saw this problem when trying to import the QuTiP toolbox in a Jupyter notebook, but it's easy to replicate in a command prompt.)

Answer: The pointer that Jay provided provided the correct clue: I needed to also add C:\Users\ray\anaconda3\envs\qutip-env\Library\lib to my PATH. I don't know how general this solution is for Python-related DLL load failures on Windows.
Title: When using autosave with black and isort in vscode, isort does not work
Tags: python,autosave,isort

Question: I want to use autosave in Visual Studio Code and apply commonly used tools like flake8, mypy, isort, and black.
Flake8, mypy, and black work fine, but isort doesn't work at all.
Is there a way to solve this problem?
My settings.json file looks like this:
{
  "python.terminal.activateEnvInCurrentTerminal": false,
  "python.defaultInterpreterPath": "${workspaceFolder}\\.venv\\Scripts\\python.exe",
  "python.linting.enabled": true,
  "python.linting.lintOnSave": true,
  "python.linting.pylintEnabled": false,
  "python.linting.flake8Enabled": true,
  "python.linting.flake8Path": "${workspaceFolder}\\.venv\\Scripts\\pflake8.exe",
  "python.linting.mypyEnabled": true,
  "python.formatting.provider": "black",
  "[python]": {
    "editor.codeActionsOnSave": {
      "source.organizeImports": true
    },
    "editor.formatOnSave": true
  }
}

Answer: You can try adding the following line to your settings.json file:
"python.sortImports.path": "${workspaceFolder}\\.venv\\Scripts\\isort.exe"
After that, save the settings.json file and restart VS Code.
Hope this answer can help you
Title: Is there any way to highly compress PDF files through python?
Tags: django,pdf,compression,wkhtmltopdf,python-pdfkit

Question: I have generated a PDF file through python using pdfkit library running on django server.
The size of pdf generated is 43.2 MB.
Total pages in pdf = 15.
Each page has 70 images, each image size = 623 bytes.
Tech Stack version used-
-Python = 3.8.16
-pdfkit = 1.0.0
-wkhtmltopdf = 0.12.6
-Django = 3.2.16
System OS = Ubuntu 22.04.2 LTS
Requirement is to compress this pdf file without compromising with the quality of content, images in it.
Any approach or suggestions for improvements in the things tried?
Things tried:

PDFNetPython3 python package- requires license/key file.
Ghost-script command from system(ubuntu) terminal to reduce file.
shrinkpdf.sh - file got from github.
ps2pdf command from system terminal.
pdfsizeopt
cpdf
Making zip of original file
Changing dpi value in pdfkit options.

File size after compression with quality of content maintained = 23.7 MB.
Expectation is have more reduction in file size.

Answer: I had a similar problem - one report generated with pdfkit was resulting in a 32mb file that I couldn't deliver through discord.
I ended up changing my code to create the report using fpdf2 and the report now is less than 2mb
Title: In Python, when to modify self and when to return new instance?
Tags: python,class,instance

Question: I am very often in a great dilemma, when should a class modify itself and when to return a new modified instance.
Lets say I want to program a simple sequence library.
class Seq:
   def __init__(self, seq):
      self.seq = seq

and then I want to rename the elements in the sequence, there are two options for this method in Seq:
def rename(lookup):
   self.seq = [lookup[e] for e in self.seq]

or
def rename(lookup):
   return Seq([lookup[e] for e in self.seq])

I will be dealing with more complicated structures, for exampke, if I am creating a graph and I want to put it into unique canonical (unique vertex enumeration) form, should it put itself in this form or return a new instance. What are most common practices about this and when to choose the two options?
My thinking is this: you have to participate what the users will have most use of. But is this
def rename(lookup, create_new_instance = False):
   if create_new_instance:
      return [lookup[e] for e in self.seq]
   else:
      self.seq = [lookup[e] for e in self.seq]

a good practice?
Or should I just implement two different methods, one as a static, that returns new object and one non-static that modifies itself? Like list.sort() and sorted(list)? But then there is a lot of overhead if I have a lot of methods that modify the object.

Answer: Typically, for built-ins the rule is:

If the class instances are logically mutable, and the method makes sense as a mutation method, only make mutating methods (and they always return None, so there is no confusion as to whether they returned themselves mutated, or a mutated copy)

If the class instances are logically immutable, you have no choice, all such "mutating" methods return new objects (note: New objects of that type typically; rename returning an unrelated list instead of a new Seq would be weird)


Do not have methods that vary in the return like that based on the argument, that's pointlessly confusing, especially if people pass it positionally; quick, what does obj.rename(xyz, True) mean as opposed to obj.rename(xyz, False)? It's more work for you, and more work for them, just don't do it.
Since your class is logically mutable, follow the rules for #1. Don't make a bunch of variant methods though, just define slicing (so they can do myseq[:] for a shallow copy) and/or a copy method (list provides both, you may as well). If the user wants to preserve the original thing, they copy first, then mutate the result. Makes it much clearer in use, and avoids massive code duplication.
Title: In Python, when to modify self and when to return new instance?
Tags: python,class,instance

Question: I am very often in a great dilemma, when should a class modify itself and when to return a new modified instance.
Lets say I want to program a simple sequence library.
class Seq:
   def __init__(self, seq):
      self.seq = seq

and then I want to rename the elements in the sequence, there are two options for this method in Seq:
def rename(lookup):
   self.seq = [lookup[e] for e in self.seq]

or
def rename(lookup):
   return Seq([lookup[e] for e in self.seq])

I will be dealing with more complicated structures, for exampke, if I am creating a graph and I want to put it into unique canonical (unique vertex enumeration) form, should it put itself in this form or return a new instance. What are most common practices about this and when to choose the two options?
My thinking is this: you have to participate what the users will have most use of. But is this
def rename(lookup, create_new_instance = False):
   if create_new_instance:
      return [lookup[e] for e in self.seq]
   else:
      self.seq = [lookup[e] for e in self.seq]

a good practice?
Or should I just implement two different methods, one as a static, that returns new object and one non-static that modifies itself? Like list.sort() and sorted(list)? But then there is a lot of overhead if I have a lot of methods that modify the object.

Answer: It mainly depends on what you want the user to be aware of in your structure.
On one hand, modifying self.seq ensures that you keep entire control of your structure, because you can control how this field is accessed (through properties for example)
On the other hand, returning the collection directly allows the user to make any transformation on it, so you will not keep control of what appends to seq. It can entirely break you class logic if the right checks are not done.
Title: Exception: Reindexing only valid with uniquely valued Index objects - indexes are unique
Tags: python,pandas

Question: I am attempting to concat two dataframes. When I tried doing so, I was hit with the error that the Reindexing only valid with uniquely valued Index objects while trying to run this code:
df_3=pd.concat([df_1,df_2]).
I have added the following checks and updates to try and identify the issue:
df_1= df_1.reset_index(drop=True)
df_2= df_2.reset_index(drop=True)
try:
    df_3=pd.concat([df_1,df_2], ignore_index=True)

except Exception as e:
    print(df_1.index.is_unique)
    print(df_1.index.to_list())
    print(df_2.index.is_unique)
    print(df_2.index.to_list())
    df_1.to_csv("../../df_1.csv")
    df_2.to_csv("../../df_2.csv")
    raise Exception(e)

This results in the following output:
True
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44,
 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 8
7, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123,
 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140]
True
[0, 1]
Reindexing only valid with uniquely valued Index objects

I do not know why this is occurring - I do not get the same issue when I am testing in a console. Please advise!

Answer: Error was I has duplicate column names - two columns in one dataframe with the same value. Pandas throws the same error in that situation. Thank you to @Timeless for finding the issue!