eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How do I test that my program is robust to unexpected shut-downs? My python code will run on a microcontroller that shuts off unexpectedly. I would like to test each part of the code rebooting unexpectedly and verify that it handles this correctly. Attempt: I tried putting code into its own process, then terminating it early, but this doesn't work because MyClass calls 7zip from the command line which continues even after process dies: import multiprocessing import os def MyClass(multiprocessing.Process): ... def run(): os.system("7z a myfile.7z myfile") process = MyClass() process.start() time.sleep(4) print("terminating early") process.terminate() print("done")  What I want: class TestMyClass(unittest.TestCase): def test_MyClass_continuity(self): myclass = MyClass().start() myclass.kill_everything() myclass = MyClass().start() self.assert_everything_worked_as_expected()  Is there an easy way to do this? If not, how do you design robust code that could terminate at any point (e.g. testing state machines)? Similar question (unanswered as of 26/10/21): Simulating abnormal termination in pytest Thanks a lot!

Answer: Your logic starts a process wrapped within the MyClass object which itself spawns a new process via the os.system call. When you terminate the MyClass process, you kill the parent process but you leave the 7zip process running as orphan. Moreover, the process.terminate method sends a SIGTERM signal to the child process. The child process can intercept said signal and perform some cleanup routines before terminating. This is not ideal if you want to simulate a situation where there is no chance to clean up (a power loss). You most likely want to send a SIGKILL signal instead (on Linux). To kill the parent and child process, you need to address the entire process group. import os import time import signal import multiprocessing class MyClass(multiprocessing.Process): def run(self): # Ping localhost for a limited amount of time os.system("ping -c 12 127.0.0.1") process = MyClass() process.start() time.sleep(4) print("terminating early") # Send SIGKILL signal to the entire process group group_id = os.getpgid(process.pid) os.killpg(group_id, signal.SIGKILL) print("done")  The above works only on Unix OSes and not on Windows ones. For Windows, you need to use the psutil module. import os import time import multiprocessing import psutil class MyClass(multiprocessing.Process): def run(self): # Ping localhost for a limited amount of time os.system("ping -c 12 127.0.0.1") def kill_process_group(pid): process = psutil.Process(pid) children = process.children(recursive=True) # First terminate all children for child in children: child.kill() psutil.wait_procs(children) # Then terminate the parent process process.kill() process.wait() process = MyClass() process.start() time.sleep(4) print("terminating early") kill_process_group(process.pid) print("done")
Question: I have the following enum in python: class Status(enum.Enum): produced = 1 consumed = 2 success = 3 failed = 4  and the following SQLAlchemy model class Message(Base): __tablename__ = 'table' id = Column(Integer, primary_key=True) name = Column(String) status = Column(Enum(Status))  I want to filter on several statuses in query like this: session.query(Message).filter(Message.status in [Status.failed, Status.success])  But no matter what I have in my DB, the results are always empty, it's probably because don't understand the type of Message.status. However, this does work: session.query(Message).filter(Message.status == Status.failed or Message.status == Status.success) 

Answer: You should use in_ operator: session.query(Message).filter(Message.status.in_((Status.failed, Status.success)))
Question: I have followed the instructions to install Brownie on Visual studio code of their website. python3 -m pip install --user pipx python3 -m pipx ensurepath  The 2 lines above poses no problem. I restarted the terminal to input line: pipx install eth-brownie  pipx : The term 'pipx' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. Was wondering what went wrong. Any form of assistance is greatly appreciated.

Answer: Check your environment variables. python3 -m pipx ensurepath adds the directories to PATH, but they are added in all lowercase for some reason. Fix the directory paths to match the case.
Question: Using Python SQLModel, I want to truncate a table, or remove all rows, and get the number of deleted rows, in the most SQLModel standar way. How can I do this? I am using this:  with Session(engine) as session: count = session.exec(delete(MyModel)).fetchall() session.commit()  But it raises an error: ResourceClosedError('This result object does not return rows. It has been closed automatically.')  I have also tried scalar() and fetchone() instead of fetchall() without success.

Answer: with Session(engine) as session: statement = delete(MyModel) result = session.exec(statement) session.commit() print(result.rowcount)  Matched Row Counts
Question: I am attempting to execute a raw sql insert statement in Sqlalchemy, SQL Alchemy throws no errors when the constructed insert statement is executed but the lines do not appear in the database. As far as I can tell, it isn't a syntax error (see no 2), it isn't an engine error as the ORM can execute an equivalent write properly (see no 1), it's finding the table it's supposed to write too (see no 3). I think it's a problem with a transaction not being commited and have attempted to address this (see no 4) but this hasn't solved the issue. Is it possible to create a nested transaction and what would start the 'first' so to speak? Thankyou for any answers. Some background:  I know that the ORM facilitates this and have used this feature and it works, but is too slow for our application. We decided to try using raw sql for this particular write function due to how often it's called and the ORM for everything else. An equivalent method using the ORM works perfectly, and the same engine is used for both, so it can't be an engine problem right?  I've issued an example of the SQL that the method using raw sql constructs to the database directly and that reads in fine, so I don't think it's a syntax error.  it's communicating with the database properly and can find the table as any syntax errors with table and column names throw a programmatic error so it's not just throwing stuff into the 'void' so to speak.  My first thought after reading around was that it was transaction error and that a transaction was being created and not closed, and so constructed the execute statement as such to ensure a transaction was properly created and commited.  with self.Engine.connect() as connection: connection.execute(Insert_Statement) connection.commit   The so called 'Insert Statement' has been converted to text using the sqlalchemy 'text' function, I don't quite understand why it won't execute if I pass the constructed string directly to the execute statement but mention it in case it's relevant.   Other things that may be relevant: Python3 is running on an individual ec2 instance the postgres database on another. The table in particular is a timescaledb hypertable taking realtime data, hence the need for very fast writes, but probably not relevant. Currently using pg8000 as dialect for no particular reason other than psycopg2 was throwing errors when trying the execute an equivalent method using the ORM.

Answer: Just so this question is answered in case anyone else ends up here: The issue was a failure to call commit as a method, as @snakecharmerb pointed out. Gord Thompson also provided an alternate method using 'begin' which automatically commits rather than connection which is a 'commit as you go' style transaction.
Question: dict keeps insertion order since Python 3.6 (see this). OrderedDict was developed just for this purpose (before Python 3.6). Since Python 3.6, is the key order always the same for dict or OrderedDict? I wonder whether I can do this in my code and have always the same behavior (except of equality, and some extended methods in OrderedDict) but more efficiently: if sys.version_info[:2] >= (3, 6): OrderedDict = dict else: from collections import OrderedDict  Or phrased differently, for Python >=3.6, is there any reason to use OrderedDict?

Answer: Both OrderedDict and dict are insertion-ordered¹ for iteration. There is practically no reason to use OrderedDict if iteration order is the only deciding point, especially if re-ordering is not needed. Obviously, if comparison order is desired OrderedDict and dict are not interchangeable.  Or phrased differently, for Python >=3.6, is there any reason to use OrderedDict?  These days OrderedDict is to dict what deque is to list, basically. OrderedDict/deque are based on linked lists² whereas dict/list are based on arrays. The former have better pop/move/FIFO semantics, since items can be removed from the start/middle without moving other items. Since arrays are generally very cache friendly, the linked list advantage only comes into play for very large containers. Also, OrderedDict (unlike deque) does not have guarantees for its linked list semantics and its advantage may thus not be portable. OrderedDict should primarily be used if many pop/move/FIFO operations are needed and benchmarking can compare the performance of dict vs. OrderedDict in practice.  ¹This applies to all currently supported implementations compliant with the Python language spec, i.e. CPython and PyPy since Python 3.6. ²OrderedDict in CPython still preserves O(1) key access. This is realised by also having a "regular" lookup table, using the linked list for order between items and the lookup table for direct item access. It's complicated.
Question: Suppose you have either two arrays: index = [1, 2, 3] counts = [2, 3, 2]  or a singular array arr = [1, 1, 2, 2, 2, 3, 3]  How can I efficiently construct the matrix [ [1, 1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0], [0, 0, 2, 2, 2, 0, 0], [0, 0, 2, 2, 2, 0, 0], [0, 0, 2, 2, 2, 0, 0], [0, 0, 0, 0, 0, 3, 3], [0, 0, 0, 0, 0, 3, 3] ]  with NumPy? I know that square = np.zeros((7, 7)) np.fill_diagnol(square, arr) # see arr above  produces [ [1, 0, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0, 0], [0, 0, 2, 0, 0, 0, 0], [0, 0, 0, 2, 0, 0, 0], [0, 0, 0, 0, 2, 0, 0], [0, 0, 0, 0, 0, 3, 0], [0, 0, 0, 0, 0, 0, 3] ]  How do I "expand" the diagonal by n where n is counts[index-1] for the values specified by index[I] tmp = np.array((arr * N)).reshape((len(arr), len(arr)) np.floor( (tmp + tmp.T) / 2 ) # <-- this is closer array([[1., 1., 1., 1., 1., 2., 2.], [1., 1., 1., 1., 1., 2., 2.], [1., 1., 2., 2., 2., 2., 2.], [1., 1., 2., 2., 2., 2., 2.], [1., 1., 2., 2., 2., 2., 2.], [2., 2., 2., 2., 2., 3., 3.], [2., 2., 2., 2., 2., 3., 3.]])  This gets what I want, but probably doesn't scale that well? riffled = list(zip(index, counts)) riffled # [(1, 2), (2, 3), (3, 2)]  a = np.zeros((len(arr), len(arr))) # 7, 7 square last = 0 # <-- keep track of current sub square for i, c in riffled: a[last:last+c, last:last+c] = np.ones((c, c)) * i last += c # <-- shift square  yield array([[1., 1., 0., 0., 0., 0., 0.], [1., 1., 0., 0., 0., 0., 0.], [0., 0., 2., 2., 2., 0., 0.], [0., 0., 2., 2., 2., 0., 0.], [0., 0., 2., 2., 2., 0., 0.], [0., 0., 0., 0., 0., 3., 3.], [0., 0., 0., 0., 0., 3., 3.]]) 

Answer: Try broadcasting: idx = np.repeat(np.arange(len(counts)), counts) np.where(idx==idx[:,None], arr, 0) # or # arr * (idx==idx[:,None])  Output; array([[1, 1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0, 0], [0, 0, 2, 2, 2, 0, 0], [0, 0, 2, 2, 2, 0, 0], [0, 0, 2, 2, 2, 0, 0], [0, 0, 0, 0, 0, 3, 3], [0, 0, 0, 0, 0, 3, 3]])
Question: I installed anaconda and spyder came with the installation. Spyder 4.2.5 came with the installation and I got a pop up notification that spyder=5.1.5 is available. I tried  conda update anaconda   conda install spyder=5.1.5  and gets an error: Solving environment: failed with initial frozen solve. Retrying with flexible solve. I tried letting it run for more than 8 hours, but I had to cancel it because I got tired. Tried  conda install anaconda spyder=5.1.5  and gets another error: `Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. PackagesNotFoundError: The following packages are not available from current channels:  ananconda  Current channels:  https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch  To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org  and use the search bar at the top of the page. Can someone please help on how to solve this?

Answer: (Spyder maintainer here) Our regular instructions to update Spyder don't work in this case because there are some incompatible dependencies between Spyder 5.0.5 and 5.1.5. To workaround this problem, you need to close Spyder and run the following commands in the Anaconda Prompt (or your system terminal on Linux or macOS): conda remove spyder conda remove python-language-server conda update anaconda conda install spyder=5.1.5  The second or third commands (i.e. conda remove python-language-server or conda update anaconda) could give you errors, but that's fine. Simply ignore them and continue with the other commands.
Question: I have a list of 10 dataframes named d0, d1, d2,...d9. All have 3 columns and 100 rows. d0.info() <class 'pandas.core.frame.DataFrame'> RangeIndex: 100 entries, 0 to 99 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 0 100 non-null float64 1 1 100 non-null float64 2 2 100 non-null float64 dtypes: float64(3) memory usage: 2.5 KB  I want to merge all dataframes so that I can have 3 columns and 1000 rows and then convert it into an array. s1=[d0,d1,d2,d3,d4,d5,d6,d7,d8,d9] s2=pd.concat([s1])  The above code throws error: TypeError: cannot concatenate object of type '<class 'list'>'; only Series and DataFrame objs are valid type(s1) list  I used the solution suggested in pd.concat in pandas is giving a TypeError: cannot concatenate object of type '<class 'str'>'; only Series and DataFrame objs are valid ; however, got the above error.

Answer: s1 is already a list. Doing what you did called pd.concat with a list of a list with DataFrames, which pandas doesn't allow. You should do it like this instead: s2=pd.concat(s1)
Question: Pandas 1.1.4 MRE: df = pd.DataFrame({"Code":[1,2], "view_A":[3000, 2300], "click_A":[3, 23], "view_B":[1200, 300], "click_B":[5, 3]}) df.set_index("Code", inplace=True) >>> view_A click_A view_B click_B Code 1 3000 3 1200 5 2 2300 23 300 3  Want to make it into  view click Code type 1 A 3000 3 2 A 2300 23 1 B 1200 5 2 B 300 3  I can do it, but want to explore more (clean) options. My sol'tn a_df = df[["view_A", "click_A"]].rename(columns={"view_A":"view", "click_A":"click"}) a_df["type"] = "A" b_df = df[["view_B", "click_B"]].rename(columns={"view_B":"view", "click_B":"click"}) b_df["type"] = "B" final_df = pd.concat([a_df, b_df])  But code is dirty.

Answer: This is essentially a reshape operation using stack df.columns = df.columns.str.split('_', expand=True) df.stack().rename_axis(['code', 'type'])    click view code type 1 A 3 3000 B 5 1200 2 A 23 2300 B 3 300
Question: I was wondering if it is possible to use the output of a rule directly as the input of the next rule, without having to specify the path again. I thought maybe something like this would work, but it does not in my tests: rule A: input: in_file = "path/to/in_file" output: out_file = "path/to/out_file" shell: "...." rule B: input: in_file = A.output.out_file # reference the output of rule A doesnt work like this # in_file = "path/to/out_file" -> this works but is less elegant i think output: out_file = "path/to/out_file" shell: "...."  Any help or insights are appreciated! Cheers!

Answer: Maybe this is the syntax you are looking for: rule B: input: in_file = rules.A.output.out_file, ...  Although I prefer to hardcode the filenames since it makes the script more readable.
Question: I have a model like this class Maca(models.Model): created_at = models.DateTimeField( auto_now_add=True )  Now I want in the views.py file to get all the entries that have created today, I'm trying this Maca.objects.filter(created_at=datetime.today().date())  But this looks for the clock that object is created too. P.S I can't change the field in model because I need the clock too in other purposes. Can someone help me to select all entries that have been created today? Thanks in advance

Answer: You have to just write a valid filter like this : from datetime import datetime today = datetime.today() year = today.year month = today.month day = today.day meca = Meca.objects.filter(created_at__year=year, created_at__month=month, created_at__day=day)
Question: I have a monorepo structured like this: myRepo/ ├─ project_1/ │ ├─ .venv/ │ ├─ main.py ├─ project_2/ │ ├─ .venv/ │ ├─ main.py ├─ .gitignore ├─ README.md   Can VS Code handle multiple python venvs in subfolders?  After some googling I managed a find one solution, but its not very elegant. I created a workspace and added the folders /project_1 and /project_2, that worked and I can easily switch and select Python Interpreter. I also need to modify files in /root from time to time so I added this folder as well. All this makes the Explorer folder structure bigger and more cluttered with duplicates of itself: workspace/ ├── myRepo/ │ ├── project_1/ │ │ ├── .venv/ │ │ └── main.py │ ├── project_2/ │ │ ├── .venv/ │ │ └── main.py │ ├── .gitignore │ ├── README.md │ └── myRepo/ ├── project_1/ │ ├── .venv/ │ └── main.py └── project_2/ ├── .venv/ └── main.py 

Answer: VS Code has a list of places, where it looks for virtual environments. Only environments located directly under the workspace are picked up automatically. You can also enter custom paths when running the Python: Select Interpreter command, though. Simply select "Enter interpreter path..." and navigate to your venv's /bin/python executable:     Once you have used a cutom interpreter path, it is known to VS code and will be directly selectable using the Python: Select Interpreter command.
Question: I'm trying to compile and install Python 3.10 into a Amazon Linux 2 but I'm not being able to get it with https support. Here the commands that I'm using to compile it: sudo yum -y update sudo yum -y groupinstall "Development Tools" sudo yum -y install openssl-devel bzip2-devel libffi-devel wget https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tgz tar xzf Python-3.10.0.tgz cd Python-3.10.0 sudo ./configure --enable-optimizations sudo make altinstall  The binary works, but when I try to use it with for reach an https endpoint, I get this message: Traceback (most recent call last): File "<stdin>", line 1113, in <module> File "<stdin>", line 1087, in main File "/usr/local/lib/python3.10/urllib/request.py", line 216, in urlopen return opener.open(url, data, timeout) File "/usr/local/lib/python3.10/urllib/request.py", line 519, in open response = self._open(req, data) File "/usr/local/lib/python3.10/urllib/request.py", line 541, in _open return self._call_chain(self.handle_open, 'unknown', File "/usr/local/lib/python3.10/urllib/request.py", line 496, in _call_chain result = func(*args) File "/usr/local/lib/python3.10/urllib/request.py", line 1419, in unknown_open raise URLError('unknown url type: %s' % type) urllib.error.URLError: <urlopen error unknown url type: https>  I'm not sure what I'm missing :/

Answer: From Python3.10, OpenSSL 1.1.1 or newer is required. (Ref: PEP 644, Python3.10 release) I have tried changing some of your code like below and worked. delete: openssl-devel add: openssl11 openssl11-devel.
Question: I have a dataframe, df, where I would like to rename two duplicate columns in consecutive order: Data DD Nice Nice Hello 0 1 1 2  Desired DD Nice1 Nice2 Hello 0 1 1 2  Doing df.rename(columns={"Name": "Name1", "Name": "Name2"})  I am running the rename function, however, because both column names are identical, the results are not desirable.

Answer: You could use an itertools.count() counter and a list expression to create new column headers, then assign them to the data frame. For example: >>> import itertools >>> df = pd.DataFrame([[1, 2, 3]], columns=["Nice", "Nice", "Hello"]) >>> df Nice Nice Hello 0 1 2 3 >>> count = itertools.count(1) >>> new_cols = [f"Nice{next(count)}" if col == "Nice" else col for col in df.columns] >>> df.columns = new_cols >>> df Nice1 Nice2 Hello 0 1 2 3  (Python 3.6+ required for the f-strings) EDIT: Alternatively, per the comment below, the list expression can replace any label that may contain "Nice" in case there are unexpected spaces or other characters: new_cols = [f"Nice{next(count)}" if "Nice" in col else col for col in df.columns]
Question: Say we have this >>> x = {'a': 1, 'b': 2} >>> y = {} >>> for k, y[k] in x.items(): pass ... >>> y {'a': 1, 'b': 2}  Why does this work? Note: I saw this first here

Answer: a, b = (c, d) unpacks the tuple from left to right and assigns a = c and b = d in that order. x.items() iterates over key-value pairs in x. E.g. doing list(x.items()) will give [('a', 1), ('b', 2)] for a, b in x.items() assigns the key to a, and the value to b for each key-value pair in x. for k, y[k] in x.items() assigns the key to k, and the value to y[k] for each key-value pair in x. You can use k in y[k] because k has already been assigned since unpacking happens left-right You don't need to do anything in the loop because whatever you needed is done already. Because the loop already assigned every value in x to y[k], y is now a shallow copy of x. As the tweet you reference says, this is indeed a "terse, unintuitive, and confusing" way to do x.copy()