Myashka/SO-Python_QA-filtered-2023-tanh_score-after_2023_02 huggingface dataset:
Title: Difference between bare except and specifying a specific exception
Tags: python,exception,try-catch

Question: In this piece of code, I could write a simple except clause without writing Exception in front of it. I mean  the last line could be like this :
except:
  print('Hit an exception other than KeyError or NameError!')

What is the point of writing Exception in front of an except clause ?
try:
    discounted_price(instrument, discount)
except KeyError:
    print("There is a keyerror in your code")
except NameError:
    print('There is a TypeError in your code')
except Exception:
    print('an exception occured')

I tried writing an except clause without Exception keyword and it worked the same.
Thank you guys for your answers . I know the point of catching specific errors. If I want to ask more clearly , what is the difference between two clauses :
except Exception:
    print('an exception occured')

except :
    print('an exception occured')

Answer: The purpose of writing "Exception" in front of an except clause is to catch all possible exceptions that can occur in the code. By specifying "Exception", you are telling the interpreter to handle any type of exception that might be raised. The more specific the exception specified in the except clause, the more targeted the handling of the exception can be. For example, if you only want to handle "KeyError" exceptions, you can specify that explicitly in the except clause, as in the first example.
Title: what is the difference between object detection fps and video fps?
Tags: python,deep-learning,frame-rate

Question: `I'm using in edge impulse FOMO
I know that object detection fps is 1/inference time
my model's time per inference is 2ms
so object detection is 500fps
but my model run vscode fps is 9.5
what is the difference between object detection fps and video fps ?

Answer: If I understand correctly, your object detection fps indicates the number of frames (or images) that your model, given your system, can process in a second.
A video fps in your input source's frames per second. For example, if your video has an fps (also referred to as framerate) of 100, then your model would be able to detect objects in all of the frames in 100ms (or 1/10 of a second).
In your case, your video input source seems to have 9.5 frames in a second. This means that your model, given your system, will process 1-second wort of a video in about ~20ms.
Title: How to identify python in windows 10
Tags: python

Question: Recently I have install python 3.9.9 in my windows 10.it want show the path
I have typed cmd promt "Wchich Python" it want show

Answer: In Command Prompt, either which python or where python will print the path to your python executable.
If which python or where python does not show the path to your Python executable it is likely that it is not in your PATH variable.
To add your executable to the PATH variable you, search for Environment Variables in the Settings application. This will open the Advanced tab in System Properties. Click the Environment Variables button towards the bottom. You can then edit the PATH variable to include the path to your Python executable. Once you have applied the changes and restarted Command Prompt you can then run which python or where python to confirm your changes have taken effect.
Title: How to identify python in windows 10
Tags: python

Question: Recently I have install python 3.9.9 in my windows 10.it want show the path
I have typed cmd promt "Wchich Python" it want show

Answer: Just type python or python3 in cmd
Title: How to identify python in windows 10
Tags: python

Question: Recently I have install python 3.9.9 in my windows 10.it want show the path
I have typed cmd promt "Wchich Python" it want show

Answer: You can use in your cmd

where python

It will show you the path of all installed python in your device
Title: creating python files from pycharm terminal
Tags: python,terminal,pycharm

Question: How does one create a python file from pycharm terminal.In VS Code they use "code {name}" so I want something similar to that but in pycharm.
I am getting an error "zsh:command not found:code"

Answer: Setting->Keymap
Search "new"
Under "Pyhon Community Edition" there will be an option for "Python File"
Add a new shortcut to this option (SHIFT+N is usually unassigned)
Title: How to print up to 40 rows in DataFrame
Tags: python,pandas,dataframe

Question: I'm trying to get data from csv and output it to the console (ie, command line).
I have 30 columns, but I can only output 5 to 6 columns.
df = pd.read_csv(csv_raw)
print(df.head())
         date        level   mark       source  
0  2022-01-01     A        1         facebook
1  2022-01-01     B        2         facebook
2  2022-01-01     C        12       facebook
3  2022-01-01     D        53       facebook
4  2022-01-01     T        22       facebook

If I display all 30 columns it turns out like this:
print(df.head(30))
          date       ...    source
0   2022-01-01  ...  facebook
1   2022-01-01  ...  facebook
2   2022-01-01  ...  facebook
3   2022-01-01  ...  facebook
4   2022-01-01  ...  facebook
5   2022-01-01  ...  facebook

when i try pd.options.display.max_columns = 50
it returns me  like that:
         date                               level  clicks  \
0  2022-01-01                                 A       1   
1  2022-01-01                                 B       2   
2  2022-01-01                                 C       12   
3  2022-01-01                                 D       53   
4  2022-01-01                                 T       22   
5  2022-01-01  Free trial, upgrade to basic at https://www.wi...       1   

     source  
0   facebook  
1   facebook  
2   facebook  
3   facebook  
4   facebook  
5   facebook  

Is it possible somehow to display more than 5 columns as in the first case?

Answer: There are 3 dataframe settings to be set to display the desired output
(1) Set the overall width (number of characters)
pd.options.display.width = 500
pd.options.display.width = None    #for unlimited
(2) Set the maximum columns count (number of columns)
pd.options.display.max_columns = 50
pd.options.display.max_columns = None    #for unlimited
(3) Set the maximum width of each column (number of characters)
pd.options.display.max_colwidth = 30
pd.options.display.max_colwidth = None    #for unlimited
There is a row (index 5) having the value Free trial, upgrade to basic at https://www.wi...  which is making a mess of the columns. To delete this row, use:
df.drop(5, inplace=True)
Title: Explaining the Output of Comparison Expressions Involving Strings and Lists in Python
Tags: python,order-of-execution,in-operator

Question: Hej,
I have the following code snippet and I don't understand the output:
a = "foo"
b = "foo"
c = "bar"
foo_list = ["foo", "bar"]


print(a == b in foo_list) # True
print(a == c in foo_list) # False

---
Output: 
True
False

The first output is True.
I dont understand it because either a == b is executed first which results in True and then the in operation should return False as True is not in foo_list.
The other way around, if b in foo_list is executed first, it will return True but then a == True should return False.
I tried setting brackets around either of the two operations, but both times I get False as output:
print((a == b) in foo_list) # False
print(a == (b in foo_list)) # False
---
Output: 
False
False

Can somebody help me out?
Cheers!

Answer: Ah, thanks @Ture Pålsson.
The answer is chaining comparisons.
a == b in foo_listis equivalent to a == b and b in foo_list where a==b is True and b in foo_list is True.
If you set brackets, there will be no chaining.
Title: Error when loading pickle object from SQLite
Tags: python,pickle

Question: I have stored a class object as a pickle in an SQLite DB.
Below is code for the file pickle.py
sqlite3.register_converter("pickle", pickle.loads)
sqlite3.register_adapter(list, pickle.dumps)
sqlite3.register_adapter(set, pickle.dumps)

class F:

    a = None
    b = None

    def __init__(self) -> None:
        pass
df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})
f = F()
f.a = df
f.b = df.columns

data = pickle.dumps(f, protocol=pickle.HIGHEST_PROTOCOL)

sqliteConnection = sqlite3.connect('SQLite_Python.db')
cursor = sqliteConnection.cursor()
print("Successfully Connected to SQLite")
DATA = sqlite3.Binary(data)
sqlite_insert_query = f"""INSERT INTO PICKLES1 (INTEGRATION_NAME, DATA) VALUES ('James',?)"""

resp = cursor.execute(sqlite_insert_query,(DATA,))
sqliteConnection.commit()

After that, I am trying to fetch the pickle from the DB. The pickle is stored in a pickle datatype column which I had registered earlier on SQLite in file retrieve_pickle.py.
cur = conn.cursor()
    cur.execute("SELECT DATA FROM PICKLES1 where INTEGRATION_NAME='James'")
    df = None
    rows = cur.fetchall()
    for r in rows[0]:
        print(type(r)) #prints <class 'bytes'>
        df = pickle.loads(r)

But it gives me an error
  File "/Users/ETC/Work/pickle_work/picklertry.py", line 34, in select_all_tasks
    df = pickle.loads(r)
AttributeError: Can't get attribute 'F' on <module '__main__' from '/Users/rusab1/Work/pickle_work/picklertry.py'>

I was trying to store a class object in a pickle column in sqlite after registering pickle.loads as a pickle datatype. I kept the object successfully and was able to retrieve it from DB but when I try to load it back so that I can access the thing and attributes it gives me an error.

Answer: Pickling requires you to import the actual module which you have in the pickle. I had to import F into the 2nd file where I was loading the pickle.
Title: How can I edit a GitHub repository (for a Python package) locally and run the package with my changes?
Tags: python,github,pip

Question: I have cloned a GitHub repository that contains the code for a Python package to my local computer (it's actually on a high performance cluster). I have also installed the package with pip install 'package_name'. If I now run a script that uses the package, it of course uses the installed package and not the cloned repository, so if I want to make changes to the code, I cannot run those. Is there a way to do this, potentially with pip install -e (but I read that was deprecated) or a fork? How could I then get novel updates in the package to my local version, as it is frequently updated?

Answer: If you run an IDE like PyCharm, you can mark a folder in your project as Sources Root. It will then import any packages from that folder instead of the standard environment packages.
Title: How can I edit a GitHub repository (for a Python package) locally and run the package with my changes?
Tags: python,github,pip

Question: I have cloned a GitHub repository that contains the code for a Python package to my local computer (it's actually on a high performance cluster). I have also installed the package with pip install 'package_name'. If I now run a script that uses the package, it of course uses the installed package and not the cloned repository, so if I want to make changes to the code, I cannot run those. Is there a way to do this, potentially with pip install -e (but I read that was deprecated) or a fork? How could I then get novel updates in the package to my local version, as it is frequently updated?

Answer: In the end I indeed did use pip install -e, and it is working for now. I will figure it out once the owner of the package releases another update!
Title: import vs from import in sklearn
Tags: python,scikit-learn

Question: why does 'from sklearn.impute import SimpleImputer as si' works but '
import sklearn.impute.SimpleImputer as si'

do not work
I want to know, why this won't work. I am new to python.

Answer: You can only use import with modules.
with from ... import ... you can import variables so submodules, functions, classes, and everything else.
As SimpleImputer is not a module only the second option is availiable.

Writen a bit differently import only works in general with files, from ... import works with variables declared in the script.
Title: import vs from import in sklearn
Tags: python,scikit-learn

Question: why does 'from sklearn.impute import SimpleImputer as si' works but '
import sklearn.impute.SimpleImputer as si'

do not work
I want to know, why this won't work. I am new to python.

Answer: The reason for this is the way the Python import statement works. The first import statement imports the SimpleImputer class from the sklearn.impute module and then names it si. The second import statement tries to import a module named SimpleImputer from a module named sklearn.impute. This does not work because in Python, the import statement only allows you to import modules, not submodules.
Title: How can I display a HTML page in discord using a bot with python?
Tags: python,html,discord

Question: I'd like to display an embed  with a picture using HTML, but I couldnt find anything online using python to do it. Is that even possible? if it is I would love an explanation.
I tried searching, couldnt find anything about it.

Answer: Makes sense, sounded like a ChatGPT generated answer... AFAIK you can't get HTML embeds on Discord (which is a pain in the arse as it could make the experience much more enjoyable for bot users). One way you could tackle this though is by generating a picture of the content you want to be sent stored on a server you have access to and have the bot share said picture. Lots of cons here, but that's the best I figured out. Good luck.
Title: Check whether timezone is dateutil.tz instance
Tags: python,timezone,pytz,python-dateutil

Question: There are several Python packages that implement the datetime.tzinfo interface, including pytz and dateutil.  If someone hands me a timezone object and wants me to apply it to a datetime, the procedure is different depending on what kind of timezone object it is:
def apply_tz_to_datetime(dt: datetime.datetime, tz: datetime.tzinfo, ambiguous, nonexistent):
    if isinstance(tz, dateutil.tz._common._tzinfo):
        # do dt.replace(tz, fold=...)
    elif isinstance(tz, pytz.tzinfo.BaseTzInfo):
        # do tz.localize(dt, is_dst=...)
    # other cases here

(The dateutil.tz case is a lot more complicated than I've shown, because there are a lot of cases to consider for non-existent or ambiguous datetimes, but the gist is always to either call dt.replace(tz, fold=...) or raise an exception.)
Checking dateutil.tz._common._tzinfo seems like a no-no, though, is there a better way?

Answer: It appears from the ratio of comments to answers (currently 9/0 = ∞), there is no available answer to the surface-level question (how to determine whether something is a dateutil.tz-style timezone object).  I'll open a feature request ticket with the maintainers of the library.
Title: AttributeError: 'Connection' object has no attribute 'connect' when use df.to_sql()
Tags: python,pandas,sqlalchemy,mysql-connector

Question: I am trying to store data retrieved from a website into MySQL database via a pandas data frame. However, when I make the function call df.to_sql(), the compiler give me an error message saying: AttributeError: 'Connection' object has no attribute 'connect'. I tested it couple times and I am sure that there is neither connection issue nor table existence issue involved. Is there anything wrong with the code itself? The code I am using is the following:
    from sqlalchemy import create_engine, text
    import pandas as pd
    import mysql.connector

    
    config = configparser.ConfigParser()
    config.read('db_init.INI')
    password = config.get("section_a", "Password")
    host = config.get("section_a", "Port")
    database = config.get("section_a", "Database")

    engine = create_engine('mysql+mysqlconnector://root:{0}@{1}/{2}'.
                           format(password, host, database),
                           pool_recycle=1, pool_timeout=57600, future=True)
    
    conn = engine.connect()
    df.to_sql("tableName", conn, if_exists='append', index = False)

The full stack trace looks like this:
Traceback (most recent call last):
  File "/Users/chent/Desktop/PFSDataParser/src/FetchPFS.py", line 304, in <module>
    main()
  File "/Users/chent/Desktop/PFSDataParser/src/FetchPFS.py", line 287, in main
    insert_to_db(experimentDataSet, expName)
  File "/Users/chent/Desktop/PFSDataParser/src/FetchPFS.py", line 89, in insert_to_db
    df.to_sql(tableName, conn, if_exists='append', index = False)
  File "/Users/chent/opt/anaconda3/lib/python3.9/site-packages/pandas/core/generic.py", line 2951, in to_sql
    return sql.to_sql(
  File "/Users/chent/opt/anaconda3/lib/python3.9/site-packages/pandas/io/sql.py", line 698, in to_sql
    return pandas_sql.to_sql(
  File "/Users/chent/opt/anaconda3/lib/python3.9/site-packages/pandas/io/sql.py", line 1754, in to_sql
    self.check_case_sensitive(name=name, schema=schema)
  File "/Users/chent/opt/anaconda3/lib/python3.9/site-packages/pandas/io/sql.py", line 1647, in check_case_sensitive
    with self.connectable.connect() as conn:

AttributeError: 'Connection' object has no attribute 'connect'

The version of pandas I am using is 1.4.4, sqlalchemy is 2.0
I tried to make a several execution of sql query, for example, CREATE TABLE xxx IF NOT EXISTS or SELECT * FROM, all of which have given me the result I wish to see.

Answer: I have faced the same problem and it got solved (As @nacho suggested above in a comment to the question) when I replace connection object with sqlalchemy engine in DataFrame.to_sql() arguments.
Title: How do I change 2 random variables out of 4?
Tags: python,variables,random,replace

Question: I need to change the value of two random variables out of four to '—'. How do I do it with maximum effectiveness and readability?
Code below is crap just for reference.
from random import choice
a = 10
b = 18
c = 15
d = 92

choice(a, b, c, d) = '—'
choice(a, b, c, d) = '—'

print(a, b, c, d)

>>> 12 — — 92
>>> — 19 — 92
>>> 10 18 — —

I've tried choice(a, b, c, d) = '—' but ofc it didn't work. There's probably a solution using list functions and methods but it's complicated and almost impossible to read, so I'm searching for an easier solution.

Answer: Variable names are not available when you run your code, so you cannot change a "random variable". Instead, I recommend that you use a list or a dictionary. Then you can choose a random element from the list or a random key from the dictionary.
Title: AccessDenied when calling Boto3 assume_role from EC2 even with service principal
Tags: python,amazon-web-services,amazon-ec2,boto3

Question: I'm running into an issue when trying to have a Python script running on an EC2 instance assume a role to perform S3 tasks.  Here's what I have done.

Created a IAM role with AmazonS3FullAccess permissions and got the following ARN:

arn:aws:iam::<account_number>:role/<role_name>
The trust policy is set so the principal is a the EC2 service.  I interpret this as allowing any EC2 instance within the account being allowed to assume the role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}



I launched an EC2 instance and attached the above IAM role.

I attempt to call assume_role() using Boto3



session = boto3.Session()
sts = session.client("sts")
response = sts.assume_role(
   RoleArn="arn:aws:iam::<account_number>:role/<role_name>",
   RoleSessionName="role_session_name"
)


But it throws the following error:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when
calling the AssumeRole operation: User:
arn:aws:sts::<account_number>:assumed-role/<role_name>/i-<instance_id>
is not authorized to perform: sts:AssumeRole on resource:
arn:aws:iam::<account_number>:role/<role_name>

All other StackOverflow questions about this talk about the Role's trust policy but mine is set to allow EC2.  So either I'm misinterpreting what the policy should be or there is some other error I can't figure out.

Answer: You do not have to explicitly call sts.assume_role. If the role is attached to the EC2 instance, boto3 will use in the background seamlessly.You use boto3 as you would normally do, and it will take care of using the IAM role for you. No action required from you.
Title: Anaconda Jupyter Notebook Opencv not working
Tags: opencv,anaconda,jupyter,nameerror,pythonpath

Question: Ok so to preface this, I am very new to jupyter notebook and anaconda. Anyways I need to download opencv to use in my notebook but every time I download I keep getting a NameError saying that ‘cv2’ is not defined.
I have uninstalled and installed opencv many times and in many different ways and I keep getting the same error. I saw on another post that open cv is not in my python path or something like that…
How do I fix this issue and put open cv in the path? (I use Mac btw) Please help :( Thank you!

Answer: Try the following:

Install OpenCV using Anaconda Navigator or via terminal by running:
conda install -c conda-forge opencv
Now you should check if its installed by running this in terminal: conda list
Import OpenCV in Jupyter Notebook: In your Jupyter Notebook, run import cv2 and see if it works.
If the above steps are not working, you should add OpenCV to your Python PATH by writing the following code to your Jupyter NB:
import sys
sys.path.append('/anaconda3/lib/python3.7/site-packages')

This should work.
Title: How to call yolov7 detect method from a python program
Tags: python,object-detection,yolo,yolov5,yolov7

Question: So far, I am using detect.py with appropiate arguments for the object detection tasks, using a custom trained model.
How can I call the detect method with the parameters(weights, source, conf, and img_size) from a python program, instead of using CLI script?
I am unable to do so.

Answer: you can create a main.py file where you call all these methods from.
Please make sure you import these methods at the top of main.py, e.g. from detect.py import detect (or whatever you want to call from this file).
Hard to give more precise advice without more input from you.
and then you just run your main file.
Alternatively maybe consider using a jupyter notebook - not the 'nicest' way, but it makes everything more convenient for testing etc.
Title: Polars DataFrame save to sql
Tags: python-polars,rust-polars

Question: Is there a way to save Polars DataFrame into a database, MS SQL for example?
ConnectorX library doesn’t seem to have that option.

Answer: Polars exposes the write_database method on the DataFrame class.
Title: Polars DataFrame save to sql
Tags: python-polars,rust-polars

Question: Is there a way to save Polars DataFrame into a database, MS SQL for example?
ConnectorX library doesn’t seem to have that option.

Answer: Polars doesen't support direct writing to a database. You can proceed in two ways:

Export the DataFrame in an intermediate format (such as .csv using .write_csv()), then import it into the database.
Process it in memory: you can convert the DataFrame in a simpler data structure using .to_dicts(). The result will be a list of dictionaries, each of them containing a row in key/value format. At this point is easy to insert them into a database using SqlAlchemy or any specific library for your database of choice.
Title: conda disappeared, command not found - corrupted .zshrc
Tags: python,macos,conda

Question: All of the sudden, my terminal stopped recognizing the 'conda'. Also the VS Code stopped seeing my environments.
All the folders, with my precious environments are there (/opt/anaconda3), but when I type conda I get:
conda          
zsh: command not found: conda

I tried install conda again (from .pkg) but it fails at the end of installation (no log provided).
How can I clean it without losing my envs?
I use Apple M1 MacBookPro with Monterey.

Answer: For some reason my .zshrc file was corrupted after some operations.
This prevented terminal to call conda init and in general, to have 'conda' call understandable.
What is more - this prevented installing any condas, minicondas, minoforge. Both from .pkg and .sh - annoyingly - without any log, information - just crash and goodbye.
I cleared both .zshrc' and .bash_profile` and then it helped - I managed to install minigorge and have my 'conda' accessible from terminal.
Unfortunately, in the process I removed all my previous 'envs'.
Title: conda disappeared, command not found - corrupted .zshrc
Tags: python,macos,conda

Question: All of the sudden, my terminal stopped recognizing the 'conda'. Also the VS Code stopped seeing my environments.
All the folders, with my precious environments are there (/opt/anaconda3), but when I type conda I get:
conda          
zsh: command not found: conda

I tried install conda again (from .pkg) but it fails at the end of installation (no log provided).
How can I clean it without losing my envs?
I use Apple M1 MacBookPro with Monterey.

Answer: To recover conda if it has disappeared and you're getting a "command not found" error, follow these steps:

Check if conda is installed on your system by running the command:
which conda

If the above command doesn't return anything, you may need to add the path to your conda installation to your PATH environment variable. To find the path, run the following command:
find / -name conda 2>/dev/null

Add the path to your .bashrc or .bash_profile file:
export PATH="/bin:$PATH"

Restart your terminal or run the following command to reload your environment variables:
source ~/.bashrc

Try running conda again to see if it's working.


If conda is still not working, it may have been uninstalled or moved. In that case, you can reinstall conda from the Anaconda website or from the Miniconda website.
Title: webbrowser library is not working as intended on SikulixIDE
Tags: python,jython,sikuli,sikuli-ide,sikuli-x

Question: On SikulixIDE, the library webbrowser always open the default browser, even when i use the get method, i tried my code on regular python, it does work. Anyone know why it is reacting like that ?
webbrowser.get('C:/Program Files/Google/Chrome/Application/chrome.exe %s').open(myurl)

Answer: Fixed by automating it using a python file and running it trough cmd with base python.exe.
Title: Python NumPy, remove unnecessary brackets
Tags: python,numpy

Question: In Numpy, Transposing of a column vector makes the the array an embedded array.
For example, transposing
[[1.],[2.],[3.]] gives [[1., 2., 3.]] and the dimension of the outermost array is 1. And this produces many errors in my code. Is there a way to produce [1., 2., 3.] directly?

Answer: Try .flatten(), .ravel(), .reshape(-1), .squeeze().
Title: ERROR: Failed building wheel for sentencepiece while installing flair on python 3.10
Tags: python,python-3.x,flair

Question: While installing flair  using pip install flair in python 3.10 virtual environment on mac-os Ventura, I get the following error:
ERROR: Failed building wheel for sentencepiece
Seperately installing sentencepeice using pip install sentenpeice did not work.
Upgrading pip did not work.

Answer: Try downgrading Python.
I was having this same issue, also with an intel mac, everytime I tried to use the transformers library, went through a lot of possible solutions without success, even with multiple chatGPT advices. I uninstalled python 3.11 and got back to the 3.9.13 version and the issue was gone! It seems there's some issue with wheels for latest python versions
Title: Manage.py unknown command
Tags: python,django,pycharm

Question: I am a student and my profesor needs me to install Django on PyCharm.
I made a big folder called PyCharmProjects and it includes like everything I have done in Python.
The problem is that I made a new folder inside this PyCharmProjects called Elementar, and I need to have the Django folders in there but it's not downloading.
I type in the PyCharm terminal django-admin manage.py startproject taskmanager1 (this is how my profesor needs me to name it)
After I run the code it says:
No Django settings specified.
Unknown command: 'manage.py'
Type 'django-admin help' for usage.
I also tried to install it through the MacOS terminal but I don't even have acces the folder named Elementar (cd: no such file or directory: Elementar) although it is created and it is seen in the PyCharm.

Answer: First of all, you can't create a project using manage.py because the manage.py file doesn't exist yet. It will be created automatically in the folder taskmanager1 if you run the command below.
You can create a project with the command
django-admin startproject taskmanager1
After that you can change the directory to the taskmanager1 folder with the cd taskmanager/ command.
When you changed the directory you can use the python manage.py commando for example if you want to run your migrations or creating an app.
python manage.py migrate
Title: How to run unittest tests from multiple directories
Tags: python,unit-testing,python-unittest

Question: I have 2 directories containing tests:
project/
|
|-- test/
|    |
|    |-- __init__.py
|    |-- test_1.py
|
|-- my_submodule/
     |
     |-- test/
          |
          |-- __init__.py
          |-- test_2.py


How can I run all tests?
python -m unittest discover .
only runs test_1.py
and obviously
python -m unittest discover my_submodule
only runs test_2.py

Answer: unittest currently sees project/my_submodule as an arbitrary directory to ignore, not a package to import. Just add project/my_submodule/__init__.py to change that.
Title: Pandas JSON Orient Autodetection
Tags: python,json,pandas

Question: I'm trying to find out if Pandas.read_json performs some level of autodetection. For example, I have the following data:
data_records = [
    {
        "device": "rtr1",
        "dc": "London",
        "vendor": "Cisco",
    },
    {
        "device": "rtr2",
        "dc": "London",
        "vendor": "Cisco",
    },
    {
        "device": "rtr3",
        "dc": "London",
        "vendor": "Cisco",
    },
]

data_index = {
    "rtr1": {"dc": "London", "vendor": "Cisco"},
    "rtr2": {"dc": "London", "vendor": "Cisco"},
    "rtr3": {"dc": "London", "vendor": "Cisco"},
}

If I do the following:
import pandas as pd
import json

pd.read_json(json.dumps(data_records))
---
  device      dc vendor
0   rtr1  London  Cisco
1   rtr2  London  Cisco
2   rtr3  London  Cisco

though I get the output that I desired, the data is record based. Being that the default orient is columns, I would have not thought this would have worked.
Therefore is there some level of autodetection going on? With index based inputs the behaviour seems more inline. As this shows appears to have parsed the data based on a column orient by default.
pd.read_json(json.dumps(data_index))

          rtr1    rtr2    rtr3
dc      London  London  London
vendor   Cisco   Cisco   Cisco

pd.read_json(json.dumps(data_index), orient="index")

          dc vendor
rtr1  London  Cisco
rtr2  London  Cisco
rtr3  London  Cisco

Answer: No, Pandas does not perform any autodetection when using the read_json function.
It is entirely determined by the orient parameter, which specifies the format of the input json data.
In your first example, you passed the data_records list to the json.dumps function, which is then converted it to a json-string. After passing the resulting json string to pd.read_json, it is seen as a  record orientation.
In your second example, you passed the data_index to json.dumps which is thenseen as a "column" orientation
In both cases, the behavior of the read_json function is entirely based on the value of the orient parameter and not by an automatic detection by Pandas.
Title: Why can't I use cdk on wsl2?
Tags: python-3.x,amazon-web-services,aws-cdk

Question: I installed cdk on wsl2 and I try to use it but I get this error:
(manifest,filePath,ASSETS_SCHEMA,Manifest.patchStackTagsOnRead)}static loadAssetManifest(filePath){return this.loadManifest(filePath,ASSETS_SCHEMA)}static saveIntegManifest(manifest,filePath){Manifest.saveManifest(manifest,filePath,INTEG_SCHEMA)}static loadIntegManifest(filePath){return this.loadManifest(filePath,INTEG_SCHEMA)}static version(){return SCHEMA_VERSION}static save(manifest,filePath){return this.saveAssemblyManifest(manifest,filePath)}static load(filePath){return this.loadAssemblyManifest(filePath)}static validate(manifest,schema4,options){function parseVersion(version){const ver=semver.valid(version);if(!ver){throw new Error(`Invalid semver string: "${version}"`)}return ver}const maxSupported=parseVersion(Manifest.version());const actual=parseVersion(manifest.version);if(semver.gt(actual,maxSupported)&&!(options==null?void 0:options.skipVersionCheck)){throw new Error(`${VERSION_MISMATCH}: Maximum schema version supported is ${maxSupported}, but found ${actual}`)}const validator=new jsonschema.Validator;const result=validator.validate(manifest,schema4,{nestedErrors:true,allowUnknownAttributes:false});let errors=result.errors;if(options==null?void 0:options.skipEnumCheck){errors=stripEnumErrors(errors)}if(errors.length>0){throw new Error(`Invalid assembly manifest:

SyntaxError: Unexpected token '?'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/aws-cdk/bin/cdk.js:3:15)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

I've tried reinstalling it, updating it, but I didn't succeed. I also searched on stack overflow but I didn't find anything to help me.

Answer: This was a problem in Node v12. Upgrading the version to v14 or higher should solve the problem.
Title: Numpy Extract Data from Compressed Sparse Column Format
Tags: python,numpy,scipy,sparse-matrix

Question: I have a mat file with sparse data for around 7000 images with 512x512 dimensions stored in a flattened format (so rows of 262144) and I’m using scipy’s loadmat method to turn this sparse information into a Compressed Sparse Column format. The data inside of these images is a smaller image that’s usually around 25x25 pixels somewhere inside of the 512x512 region , though the actual size of the smaller image is not consitant and changes for each image. I want to get the sparse information from this format and turn it into a numpy array with only the data in the smaller image; so if I have an image that’s 512x512 but there’s a circle in a 20x20 area in the center I want to just get the 20x20 area with the circle and not get the rest of the 512x512 image. I know that I can use .A to turn the image into a non-sparse format and get a 512x512 numpy array, but this option isn’t ideal for my RAM.
Is there a way to extract the smaller images stored in a sparse format without turning the sparse data into dense data?
I tried to turn the sparse data into dense data, reshape it into a 512x512 image, and then I wrote a program to find the top, bottom, left, and right edges of the image by checking for the first occurrence of data from the top, bottom, left, and right but this whole processes seemed horribly inefficient.

Answer: Sorry about the little amount of information I provided; I ended up figuring it out.Scipy's loadmat function when used to extract sparse data from a mat file returns a csc_matrix, which I then converted to numpy's compressed sparse column format. Numpy's format has a method .nonzero() that will return the index of every non_zero element in that matrix. I then reshaped the numpy csc matrix into 512x512, and then used .nonzero() to get the non-zero elements in 2D, then used used those indexes to figure out the max height and width of my image I was interested in. Then I created a numpy matrix of zeros the size of the image I wanted, and set the elements in that numpy matrix to the elements to the pixels I wanted by indexing into my numpy csc matrix (after I called .tocsr() on it)
Title: How to get the clock argument of event.post(clock, data, duration) in a python device?
Tags: python,post,events,clock,simics

Question: I cannot find an example in the Simics documentation on how the clock object is obtained so that we can use it as an argument in the post() method.
I suspect that either

an attribute can be used to get the clock or
in the ConfObject class scope we get the clock using SIM_object_clock()

I created a new module using bin\project-setup --py-device event-py
I have defined two methods in the ConfObject class scope called clock_set and clock_get.
I wanted to use these methods so that I can set/get the clock object to use in the post method.
The post() method fails when reading the device registers in the vacuum machine.
import pyobj
# Tie code to specific API, simplifying upgrade to new major version
import simics_6_api as simics


class event_py(pyobj.ConfObject):
    """This is the long-winded documentation for this Simics class.
    It can be as long as you want."""
    _class_desc = "one-line doc for the class"
    _do_not_init = object()

    def _initialize(self):
        super()._initialize()


    def _info(self):
        return []

    def _status(self):
        return [("Registers", [("value", self.value.val)])]

    def getter(self):
        return self

# In my mind, clock_set is supposed to set the clock object. That way we can use
# it in post()
    def clock_set(self):
        self.clock = simics.SIM_object_clock(self)

    def clock_get(self):
        return self.clock(self):

    class value(pyobj.SimpleAttribute(0, 'i')):
        """The <i>value</i> register."""

    class ev1(pyobj.Event):
        def callback(self, data):
            return 'ev1 with %s' % data


    class regs(pyobj.Port):
        class io_memory(pyobj.Interface):
            def operation(self, mop, info):
                offset = (simics.SIM_get_mem_op_physical_address(mop)
                          + info.start - info.base)
                size = simics.SIM_get_mem_op_size(mop)

                if offset == 0x00 and size == 1:
                    if simics.SIM_mem_op_is_read(mop):
                        val = self._up._up.value.val
                        simics.SIM_set_mem_op_value_le(mop, val)
                        # POST HERE AS TEST self._up._up.ev1.post(clock, val, seconds = 1)
                    else:
                        val = simics.SIM_get_mem_op_value_le(mop)
                        self._up._up.value.val = val
                    return simics.Sim_PE_No_Exception
                else:
                    return simics.Sim_PE_IO_Error

Answer: You mention using the vacuum example machine and within its script you see that sdp->queue will point to timer. So SIM_object_clock(sdp) would return timer.
Simics is using queue attribute in all conf-objects to reference their clock individually, though other implementations are used too.
BR
Simon
#IAmIntel
Title: ubuntu run python program in background on startup
Tags: python,background,boot

Question: I need to start a python program when the system boots. It must run in the background (forever) such that opening a terminal session and closing it does not affect the program.
I have demonstrated that by using tmux this can be done manually from a terminal session. Can the equivalent be done from a script that is run at bootup?
Then where done one put that script so that it will be run on bootup.

Answer: It appears that in addition to putting a script that starts the program in /etc/init.d, one also has to put a link in /etc/rc2.d with
sudo ln -s /etc/init.d/scriptname.sh
sudo mv scriptname.sh S01scriptname.sh
The S01 was just copied from all the other files in /etc/rc2.d
Title: How to convert key value text to json arrya format python
Tags: json,python-3.x

Question: I have a use case where we have text file like key value format .
The file is not any of the fixed format but created like key value .
We need to create JSON out of that file .
I am able to create JSON but when text format has array like structure it creates just Key value json not the array json structure .
This is my Input .
[DOCUMENT]
Headline=This is Headline
MainLanguage=EN
DocType.MxpCode=1000
Subject[0].MxpCode=BUSNES
Subject[1].MxpCode=CONS
Subject[2].MxpCode=ECOF
Author[0].MxpCode=6VL6
Industry[0].CtbCode=53
Industry[1].CtbCode=5340
Industry[2].CtbCode=534030
Industry[3].CtbCode=53403050
Symbol[0].Name=EXPE.OQ
Symbol[1].Name=ABNB.OQ
WorldReg[0].CtbCode=G4
Country[0].CtbCode=G26
Country[1].CtbCode=G2V
[ENDOFFILE]

Exiting code to create json is below
with open("file1.csv") as f:
    lines = f.readlines()
data = {}
for line in lines:
    parts = line.split('=')
    if len(parts) == 2:
        data[parts[0].strip()] = parts[1].strip()
print(json.dumps(data, indent='  '))

The current output is below
{
  "Headline": "This is Headline",
  "MainLanguage": "EN",
  "DocType.MxpCode": "1000",
  "Subject[0].MxpCode": "BUSNES",
  "Subject[1].MxpCode": "CONS",
  "Subject[2].MxpCode": "ECOF",
  "Author[0].MxpCode": "6VL6",
  "Industry[0].CtbCode": "53",
  "Industry[1].CtbCode": "5340",
  "Industry[2].CtbCode": "534030",
  "Industry[3].CtbCode": "53403050",
  "Symbol[0].Name": "EXPE.OQ",
  "Symbol[1].Name": "ABNB.OQ",
  "WorldReg[0].CtbCode": "G4",
  "Country[0].CtbCode": "G26",
  "Country[1].CtbCode": "G2V"
}

Expected out is is something like below
For the Subject key and like wise for others also
{
  "subject": [
    {
      "mxcode": 123
    },
    {
      "mxcode": 123
    },
    {
      "mxcode": 123
    }
  ]
}

Like wise for Industry and Symbol and Country.
so the idea is when we have position in the text file it should be treated as array in the json output .

Answer: Use one more loop as it is nested. Use for loop from where subject starts. try it that way.
Title: Error messages clogging Telethon resulting : Server sent a very new message xxxxx was ignored
Tags: python,telethon

Question: Whenever launching telethon from an existing session I receive two error messages:
Server sent a very new message with ID xxxxxxxxxxxxxxxxxxx, ignoring Server sent a very new message with ID xxxxxxxxxxxxxxxxxxx, ignoring
And thereafter it gets clogged , preventing any execution.
The answer I got from another post is "in Windows time settings, enable automatic setting of time and time zone". But I am using a Linux system, and the system is set to the Asia/Shanghai time zone. How can I fix this problem?

Answer: I think I found the reason. The time difference between the local environment and the Telegram server is too large. After manually adjusting the time to correct the delay, the problem was fixed.
Title: Raspberry Pi - Crontab task not running properly
Tags: python,cron,raspberry-pi,arp

Question: I have scheduled a task arp -a which runs once per hour, that scans my wi-fi network to save all the info about currently connected devices into a scan.txt file. After the scan, a python script reads the scan.txt and saves the data into a database.
This is what my wifiscan.sh script looks like:
cd /home/pi/python/wifiscan/
arp -a > /home/pi/python/wifiscan/scan.txt
python wifiscan.py

This is my crontab task:
#wifiscan
59 * * * * sh /home/pi/launcher/wifiscan.sh

If I run the wifiscan.sh file manually, all the process works perfectly; when it is run by the crontab, the scan.txt file is generated empty and the rest of the process works, but with no data, so I'm assuming that the problem lies in the arp -a command.
How is it possible that arp -a does not produce any output when it is run by crontab? Is there any mistakes I'm making?

Answer: As @Mark Setchell commented, I solved my problem by launching the command with its entire path (in this case, /usr/sbin/arp)
Title: How to test data from RDS using mock_rds
Tags: python,testing,mocking,amazon-rds,moto

Question: I am trying to test the data being written to RDS, but I can't seem to be able to mock the DB. The idea is to mock a DB, then run my code and retrieve the data for testing. Could anyone help, please?
import unittest
import boto3
import mock
from moto import mock_s3, mock_rds
from sqlalchemy import create_engine

@mock_s3
@mock_rds
class TestData(unittest.TestCase):
    def setUp(self):
        """Initial setup."""
        # Setup db

        test_instances = db_conn.create_db_instance(
            DBName='test_db',
            AllocatedStorage=10,
            StorageType='standard',
            DBInstanceIdentifier='instance',
            DBInstanceClass='db.t2.micro',
            Engine='postgres',
            MasterUsername='postgres_user',
            MasterUserPassword='p$ssw$rd',
            AvailabilityZone='us-east-1',
            PubliclyAccessible=True,
            DBSecurityGroups=["my_sg"],
            VpcSecurityGroupIds=["sg-123456"],
            Port=5432
        )
        db_instance = test_instances["DBInstance"]

        user_name = db_instance['MasterUsername']
        host = db_instance['Endpoint']['Address']
        port = db_instance['Endpoint']['Port']
        db_name = db_instance['DBName']
        conn_str = f'postgresql://{user_name}:p$ssw$rd@{host}:{port}/{db_name}'
        print(conn_str)
        engine_con = create_engine(conn_str)
        engine_con.connect()

Error:
>       conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
E       sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "instance.aaaaaaaaaa.eu-west-1.rds.amazonaws.com" to address: nodename nor servname provided, or not known
E       
E       (Background on this error at: https://sqlalche.me/e/14/e3q8)

Answer: So, instead of testing the data from my DB, I replicated the execution of the code I had in my lambda on my test, accessing the results locally. So those same tests are working fine on Github now.
Title: Remove private PyPi index from local virtual env
Tags: python,pip,pypi

Question: I have somehow managed to mess up my pip indexes for a local virtual env.
pip config list returns the following
:env:.index-url='https://***/private-pypi/simple/' global.index-url='https://pypi.python.org/simple'
This makes pip to always default to searching the private pypi index first. Any idea how I can remove the env specific index? It does not appear in the pip.conf file and running pip config unset env.index-url does not work either or I can't get the right syntax.
Thanks!

Answer: You can remove the environment-specific index by directly editing the environment's pip.ini file or pip.conf file. The file should be located in the environment's lib/pythonX.X/site-packages/pip/ directory. Simply delete the line with the "index-url" value and the default global index will be used.
Title: Creating Virtual Environment with Python 3.11 Returns an Error
Tags: python-venv,python-3.11

Question: I got an error when creating virtualenv with Python 3.11 interpreter.
I typed this in my terminal
python3.11 -m venv env

It returned this:
Error: Command '['/home/bambang/env/bin/python3.11', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

What's possibly missing?

Answer: I tried to add --without-pip flag. It doesn't return an error so far
Title: Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location
Tags: python,python-3.x,terminal,pip,zsh

Question: What does it mean when I keep getting these warnings
WARNING: The script jupyter-trust is installed in '/Users/josephchoi/Library/Python/3.9/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
I am on MacOS and zsh. I tried researching but the texts were too complicated. As you can tell, I am a complete beginner.

Answer: This normally will happen when you've installed a pip package that contains an executable, and it shouldn't be a problem. If you don't like the warning, you can add the folder to your PATH variable by adding the line export PATH=$PATH:/Users/josephchoi/Library/Python/3.9/bin to your .zshrc file in your home directory and it will stop shouting at you.