eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My co-worker asked me why his code cannot run in concurrency in multi-threads. I discover the os.system function is acting weird unlike the other functions in multi-threads. I code a small demo to reproduce the problem. Appreciating somebody who can answer my question. Run like serial execution import os import time from concurrent.futures import ThreadPoolExecutor, as_completed executor = ThreadPoolExecutor() begin = time.time() for _ in as_completed([ executor.submit(os.system, cmd) for cmd in ['sleep 10', 'sleep 3', 'sleep 2'] ]): print("completed after", time.time() - begin, "seconds")  output completed after 10.005892276763916 seconds completed after 13.017478942871094 seconds completed after 15.034425258636475 seconds  Run normally Replace os.system with subprocess.call. import subprocess import time from concurrent.futures import ThreadPoolExecutor, as_completed executor = ThreadPoolExecutor() begin = time.time() for _ in as_completed([ executor.submit(subprocess.call, cmd, shell=True) for cmd in ['sleep 10', 'sleep 3', 'sleep 2'] ]): print("completed after", time.time() - begin, "seconds")  output completed after 2.019604206085205 seconds completed after 3.017033338546753 seconds completed after 10.009897232055664 seconds 

Answer: os.system is implemented by calling the standard C function system:  This is implemented by calling the Standard C function system(), and has the same limitations.  And according to the man page of the system function:  Blocking SIGCHLD while waiting for the child to terminate prevents the application from catching the signal and obtaining status from system()'s child process before system() can get the status itself.  and:  Using the system() function in more than one thread in a process or when the SIGCHLD signal is being manipulated by more than one thread in a process may produce unexpected results.  Since SIGCHLD must be blocked during the execution, calling system concurrently would not work. subprocess.Popen (the underlying class that the subprocess.call function uses) on the other hand handles fork/exec/wait without blocking SIGCHLD and can therefore be executed concurrently.
Question: I am working with Python and some json data. I am looping through my data (which are all dictionaries) and when I print the loop values to my console, I get 1 dictionary per line. However, when I do the same line of code with json.dumps() to convert my object into a string to be able to be output, I get multiple lines within the dictionary versus wanting the new line outside the dictionary. How do you add new lines after each dictionary value when looping? Code example:  def test(values, filename): with open(filename, 'w') as f: for value in values: print(json.dumps(value, sort_keys=True)) # gives me each dictionary in a new line f.write(json.dumps(value, sort_keys=True, indent=0) #gives me a new line for each key/value pair instead of after each dictionary.  Output in the console: {"first_name": "John", "last_name": "Smith", "food": "corn"} {"first_name": "Jane", "last_name": "Doe", "food": "soup"}  Output in my output file: {"first_name": "John", "last_name": "Smith", "food": "corn"}{"first_name": "Jane", "last_name": "Doe", "food": "soup"}  What code am I missing to get a new line for each dictionary value so that my output file looks the same as my console?

Answer: You can newline after each f.write(json.dumps(value, sort_keys=True, indent=0))  like this - f.write('\n')
Question: I'm trying to plot using sns.histplot on the Titanic Dataset in Kaggle's Jupyter Notebook. This is my code: sns.histplot(train, x = "Age", hue="Sex")  But it's throwing me this error:  --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-11-d14c3b5a0670> in <module> ----> 1 sns.histplot(train, x = "Age", hue="Sex") AttributeError: module 'seaborn' has no attribute 'histplot'  I have made sure to import seaborn (previous plots using sns.barplot worked fine). I'm running on Mac OS X 10.15.6, and Seaborn version 0.11.0. Could somebody point out what went wrong? Thanks in advance.

Answer: If you have a standard python installation, update with pip: pip install -U seaborn  If you are using an Anaconda distribution, at the anaconda prompt (base) environment, or activate the appropriate environment: # update all the packages in the environment conda update --all # or conda update seaborn  See Anaconda: Managing Packages
Question: I have a MelSpectrogram generated from: eval_seq_specgram = torchaudio.transforms.MelSpectrogram(sample_rate=sample_rate, n_fft=256)(eval_audio_data).transpose(1, 2)  So eval_seq_specgram now has a size of torch.Size([1, 128, 499]), where 499 is the number of timesteps and 128 is the n_mels. I'm trying to invert it, so I'm trying to use GriffinLim, but before doing that, I think I need to invert the melscale, so I have: inverse_mel_pred = torchaudio.transforms.InverseMelScale(sample_rate=sample_rate, n_stft=256)(eval_seq_specgram)  inverse_mel_pred has a size of torch.Size([1, 256, 499]) Then I'm trying to use GriffinLim: pred_audio = torchaudio.transforms.GriffinLim(n_fft=256)(inverse_mel_pred)  but I get an error: Traceback (most recent call last): File "evaluate_spect.py", line 63, in <module> main() File "evaluate_spect.py", line 51, in main pred_audio = torchaudio.transforms.GriffinLim(n_fft=256)(inverse_mel_pred) File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torchaudio/transforms.py", line 169, in forward return F.griffinlim(specgram, self.window, self.n_fft, self.hop_length, self.win_length, self.power, File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torchaudio/functional.py", line 179, in griffinlim inverse = torch.istft(specgram * angles, RuntimeError: The size of tensor a (256) must match the size of tensor b (129) at non-singleton dimension 1  Not sure what I'm doing wrong or how to resolve this.

Answer: Just for history, full code: import torch import torchaudio import IPython waveform, sample_rate = torchaudio.load("wavs/LJ030-0196.wav", normalize=True) n_fft = 256 n_stft = int((n_fft//2) + 1) transofrm = torchaudio.transforms.MelSpectrogram(sample_rate, n_fft=n_fft) invers_transform = torchaudio.transforms.InverseMelScale(sample_rate=sample_rate, n_stft=n_stft) grifflim_transform = torchaudio.transforms.GriffinLim(n_fft=n_fft) mel_specgram = transofrm(waveform) inverse_waveform = invers_transform(mel_specgram) pseudo_waveform = grifflim_transform(inverse_waveform)  And IPython.display.Audio(waveform.numpy(), rate=sample_rate)  IPython.display.Audio(pseudo_waveform.numpy(), rate=sample_rate)
Question: I am Python newbie. Trying to use this module https://pypi.org/project/sparql-client/ module.py from sparql import Service class MyModule: def my_method(self): s = Service('https://my-endpoint:8182/sparql', "utf-8", "GET") statement = """ MOVE uri:temp_graph TO uri:user_graph ADD uri:temp_graph TO uri:user_graph """.format(user_graph="http://aws.amazon.com/account-uid", temp_graph="http://aws.amazon.com/account-uid-temp") s.query(statement)  I am trying to test it test_module.py import unittest from unittest.mock import patch, Mock class TestModule(unittest.TestCase): @patch('sparql.Service', autospec=True) def test_mymethod(self, sparql_mock): sparql_instance = sparql_mock.return_value sparql_instance.query = Mock()  While running I get  File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/mock.py", line 1564, in <lambda> getter = lambda: _importer(target) File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/mock.py", line 1236, in _importer thing = __import__(import_path) File "/usr/local/lib/python3.9/site-packages/sparql.py", line 50, in <module> from base64 import encodestring ImportError: cannot import name 'encodestring' from 'base64' (/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/base64.py)  So it can not import this line https://github.com/eea/sparql-client/blob/master/sparql.py#L50 Any idea how to fix this?

Answer: The problem is caused by the version of base64 module you are running while the version of sparql you have installed is dependent on a lower version of the base64 module. The sparql is dependent on base64 version built for python3.1. encodestring() and decodestring() have since been deprecated.  Option 1 - Your best bet (if you must continue using this version of sparql) is to downgrade your version of python to 3.1 from 3.9 which is your current version. Option 2 - will be to adopt the new specifications for the current version of base64 you have installed. this will mean updating sparql and anywhere you are calling deprecated methods of base64.  If you choose to go with option 2, then open the sparql module and edit the import statement. Change from base64 import encodestring to from base64 import encodebytes and replace any occurrences of encodestring with encodebytes in your code and any module you have dependent on base64. That should solve your problem.
Question: I have a huge graph with about 5000 nodes that I made it with networkX. It takes about 30 seconds to create this graph each time I execute my script. After this relatively long time I can run my analysis like shortest_path and so on. My question is, is there any way to store the object of this graph in file or something like this and each time I run my script, networkX read that file and load all of my graph?

Answer: As of version 2.6, methods write_gpickle and read_gpickle are deprecated. Try this instead: import pickle # save graph object to file pickle.dump(G, open('filename.pickle', 'wb')) # load graph object from file G = pickle.load(open('filename.pickle', 'rb'))  Note the options 'wb' to dump and 'rb' to load in open().
Question: I have the following two async function from tornado.httpclient import AsyncHTTPClient async def get_categories(): # return a list of str # .... http = AsyncHTTPClient() resp = await http.fetch(....) return [....] async def get_details(category): # return a list of dict # .... http = AsyncHTTPClient() resp = await http.fetch(....) return [....]  Now I need to create a function to get the details (run http fetch concurrently) for all the categories and combine them together. async def get_all_details(): categories = await get_categories() tasks = list(map(lambda x: {'category': x, 'task':get_details(x)}, categories)) r = await asyncio.gather(*tasks) # error # need to return [ # {'category':'aaa', 'detail':'aaa detail 1'}, # {'category':'aaa', 'detail':'aaa detail 2'}, # {'category':'bbb', 'detail':'bbb detail 1'}, # {'category':'bbb', 'detail':'bbb detail 2'}, # {'category':'bbb', 'detail':'bbb detail 3'}, # {'category':'ccc', 'detail':'ccc detail 1'}, # {'category':'ccc', 'detail':'aaa detail 2'}, # ]  However, the list line return the error:  TypeError: unhashable type: 'dict'  The tasks has the following values: [{'category': 'aaa', 'task': <coroutine object get_docker_list at 0x000001B12B8560C0>}, {'category': 'bbb', 'task': <coroutine object get_docker_list at 0x000001B12B856F40>}, {'category': 'ccc', 'task': <coroutine object get_docker_list at 0x000001B12B856740>}]  BTW, is it a way to throttle the http fetch calls? For example, at most four fetches running at the same time.

Answer: gather accepts coroutine (or other awaitable) arguments and returns a tuple of their results in the same order. You are passing it a sequence of dicts some of whose values are coroutines. gather doesn't know what to do with that and attempts to treat the dicts as awaitable objects, which fails soon enough. The correct way to generate the list of dicts would be to pass just the coroutines to gather, await the results, and process them into a new dict: async def get_all_details(): category_list = await get_categories() details_list = await asyncio.gather( *[get_details(category) for category in category_list] ) return [ {'category': category, 'details': details} for (category, details) in zip(category_list, details_list) ]   BTW, is it a way to throttle the http fetch calls? For example, at most four fetches running at the same time.  The convenient and idiomatic way to limit parallel calls is by using a semaphore: async def get_details(category, limit): # acquiring the semaphore passed as `limit` will allow at most a # fixed number of coroutines to proceed concurrently async with limit: ... the rest of the code ... async def get_all_details(): limit = asyncio.Semaphore(4) category_list = await get_categories() details_list = await asyncio.gather( *[get_details(category, limit) for category in category_list] ) ... the rest of the code ...
Question: I observe an inconsistent behavior when patching a dummy class: class A: def f(self, *args, **kwargs): pass  If I patch manually the function: call_args_list = [] def mock_fn(*args, **kwargs): call_args_list.append(mock.call(*args, **kwargs)) with mock.patch.object(A, 'f', mock_fn): A().f(1, 2) print(call_args_list) # [call(<__main__.A object at 0x7f0da0c08b50>, 1, 2)]  As expected mock_fn is called with the self argument (mock_fn(self, 1, 2)). However, if I'm using a mock.Mock object, somehow the self argument is removed from the call: mock_obj = mock.Mock() with mock.patch.object(A, 'f', mock_obj): A().f(1, 2) print(mock_obj.call_args_list) # [call(1, 2)]  This feel inconsistent. mock_obj is called as mock_obj(self, 1, 2), yet mock_obj.call_args == call(1, 2). It removes the self argument from call_args. How can I access the bounded-method instance ?

Answer: with mock.patch.object(A, 'f', autospec=True) as mock_obj: A().f(1, 2) print(mock_obj.call_args_list) # [call(<__main__.A object at 0x7fb2908fc880>, 1, 2)]  From the documentation:  If you pass autospec=True [...] the mocked function will be turned into a bound method if it is fetched from an instance. It will have self passed in as the first argument [...]
Question: I'm on python; I am trying to shut down a function running through a ThreadPoolExecutor, but the shutdown is crashing with the error: libgcc_s.so.1 must be installed for pthread_cancel to work  The function is submitted with: record_future = self.executor.submit(next,primitive)  primitive is an iterator that usually returns a value but in some cases, it needs to wait a while before returning a value (because of long calculations, etc). In these cases, when I need to shut down the running thread, I cannot wait for the iterator to finish returning, and need to shut it down immediately. I did it with: executor.shutdown(wait=False)  However, when execution reaches this point, I get the libgcc error. I tried 'solving' it by manually installing with: apt-get install libgcc1:amd64  but no dice. I am not sure where exactly python is looking for this library, otherwise I would try creating a symbolic link, because the library is already installed at: $ /sbin/ldconfig -p | grep libgcc libgcc_s.so.1 (libc6,x32) => /usr/libx32/libgcc_s.so.1 libgcc_s.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgcc_s.so.1 libgcc_s.so.1 (libc6) => /usr/lib32/libgcc_s.so.1 

Answer: I found a potential workaround in the Python mailing list to explicitly load libgcc_.so.1 via ctypes as follows: import ctypes libgcc_s = ctypes.CDLL('libgcc_s.so.1')  One has to make sure that this is loaded before any threads are created and that the variable libgcc_s lasts until all of the threads are closed (i.e. put this at the beginning of your file).
Question: How can I remove the undesired gap between the Python folium map and the next cell inside my jupyter notebook. Here the naive code to reproduce my problem : import folium m = folium.Map(width=600, height=400, location=[12, 12], zoom_start=2) m  

Answer: Here the solution need to use Figure : from branca.element import Figure fig = Figure(width=600, height=400) m = folium.Map(location=[12, 12], zoom_start=2) fig.add_child(m)
Question: Is there a way to show the output such as a very long data-frame in a scrollable window in VS Code Jupyter Notebook? I am aware that pressing letter "o" allows you to collapse all output. But having the scrollable window is still preferable as it allows you to check the output while referring to other windows. I also checked this link but couldn't get it work.

Answer: Per the comments, a workaround to trigger the scrollable window is to add a print statement to the cell output. For example print("foo") at the beginning or end of a cell.
Question: I ordered a MacBook Pro equipped with an M1 ARM processor. Will I be able to run Homebrew and install dev tools like Python, Node etc..?

Answer: Yes. Now Homebrew fully supports Apple Silicon https://brew.sh/2021/02/05/homebrew-3.0.0/
Question: I'm having some issues with AWS Lambda for Python 3.8. No matter what code I try running, AWS Lambda keeps returning the same response. I am trying to retrieve a information from a DynamoDB instance with the code below: import json import boto3 dynamodb = boto3.resource('dynamodb') table = dynamodb.Table('planets') def lambda_handler(event, context): response = table.get_item( Key = { 'id':'mercury' } ) print(response) # TODO implement return { 'statusCode': 200, 'body': response) }  I am expecting an output like 'body':{'Item': {'id':'mercury', 'temp':'sizzling hot'}}, or an error even, but I keep getting the response below: Response: { "statusCode": 200, "body": "\"Hello from Lambda!\"" }  I even change up the code, expecting an error, but I still get the same output.

Answer: Usually this is due to one of the following reasons:  You are not deploying your code changes. In the new UI, you have to explicitly Deploy your function using Orange button. You are invoking old lambda version, rather then your latest version, if you are versioning your functions. You must explicitly choose the correct version to invoke.
Question: I've got two models: Game and Player. I want to have a list of all players in the Game Model, as well as one of these players in a different field. It's a flask server and a sqlite database. Here is my Player Model: class Player(db.Model): id = db.Column(db.Integer, primary_key=True) # other fields... game_id = db.Column(db.Integer, db.ForeignKey('game.id')) game = db.relationship('Game', back_populates='players', foreign_keys=game_id)  Here is my Game Model: class Game(db.Model): id = db.Column(db.Integer, primary_key=True) # other fields... current_president_id = db.Column(db.Integer, db.ForeignKey('player.id')) current_president = db.relationship("Player", foreign_keys=current_president_id) # other one-to-one relationships to the Player Model, exactly like the first one... players = db.relationship('Player', back_populates='game', foreign_keys=Player.game_id)  I made the two models based on this Response. But I still get the same Warning: SAWarning: Cannot correctly sort tables; there are unresolvable cycles between tables "game, player", which is usually caused by mutually dependent foreign key constraints. Foreign key constraints involving these tables will not be considered; this warning may raise an error in a future release.  Like the Warning says, when using this configuration I then get at some point: sqlalchemy.exc.CircularDependencyError: Circular dependency detected. (SaveUpdateState(<Game at 0x23009e00160>), SaveUpdateState(<Player at 0x23009e141c0>), ProcessState(ManyToOneDP(Game.current_president), <Game at 0x23009e00160>, delete=False), ProcessState(OneToManyDP(Game.players), <Game at 0x23009e00160>, delete=False))  I have no idea what to do, I tried to read through the documentation and tested many other configurations, but nothing works. All help greatly appreciated. Thanks!

Answer: So I tried to learn SQLAlchemy a bit deeper, and found a solution. First you set the use_alter Flag to True in the one-to-one relationship: current_president_id = db.Column(db.Integer, db.ForeignKey('player.id', use_alter=True))  This makes the Warning go away. But you still need to be careful now. Because the Player model has a Column referencing the Game.id and the Game model has a Column referencing the Player.id you cannot declare the relationships in one commit: g, p1, p2 = Game(), Player(), Player() db.session.add_all([g, p1, p2]) db.session.commit() g.players = [p1, p2] g.current_president = p3 db.session.commit()  This will raise the CircularDependencyError. But this will not: g, p1, p2 = Game(), Player(), Player() db.session.add_all([g, p1, p2]) db.session.commit() g.players = [p1, p2] db.session.commit() g.current_president = p3 db.session.commit()  EDIT: Use use_alter=True in the relationship to prevent any CircularDependencyErrors. This way, when you add a Game with Players and the relationship in one commit, the relationship will be set in a second 'ALTER' statement to prevent the circular dependency.
Question: I'm using Spyder with Anaconda and since MacOS last update (Big Sur 11.0.1), when doing import geopandas, I get the following error: OSError: Could not find lib c or load any of its variants [].  There are several subjects on the matter (particularly this one and this one), that mainly recommends to reset the environment variable DYLD_FALLBACK_LIBRARY_PATH by putting in the Terminal: export DYLD_FALLBACK_LIBRARY_PATH=$(HOME)/lib:/usr/local/lib:/lib:/usr/lib  However, here is what I get: -bash: HOME: command not found  A brew or pip install didn't fix the problem either. Does anyone have an idea on how I can fix this? Am I suppose to replace $(HOME) by something? Many thanks!

Answer: For references: I ended up reinstalling a new environment on Anaconda an it works but I wasn't able to fix the original problem.
Question: I'm wondering if : a = "abcdef" b = "def" if a[3:] == b: print("something")  does actually perform a copy of the "def" part of a somewhere in memory, or if the letters checking is done in-place ? Note : I'm speaking about a string, not a list (for which I know the answer)

Answer: String slicing makes a copy in CPython. Looking in the source, this operation is handled in unicodeobject.c:unicode_subscript. There is evidently a special-case to re-use memory when the step is 1, start is 0, and the entire content of the string is sliced - this goes into unicode_result_unchanged and there will not be a copy. However, the general case calls PyUnicode_Substring where all roads lead to a memcpy. To empirically verify these claims, you can use a stdlib memory profiling tool tracemalloc: # s.py import tracemalloc tracemalloc.start() before = tracemalloc.take_snapshot() a = "." * 7 * 1024**2 # 7 MB of ..... # line 6, first alloc b = a[1:] # line 7, second alloc after = tracemalloc.take_snapshot() for stat in after.compare_to(before, 'lineno')[:2]: print(stat)  You should see the top two statistics output like this: /tmp/s.py:6: size=7168 KiB (+7168 KiB), count=1 (+1), average=7168 KiB /tmp/s.py:7: size=7168 KiB (+7168 KiB), count=1 (+1), average=7168 KiB  This result shows two allocations of 7 meg, strong evidence of the memory copying, and the exact line numbers of those allocations will be indicated. Try changing the slice from b = a[1:] into b = a[0:] to see that entire-string-special-case in effect: there should be only one large allocation now, and sys.getrefcount(a) will increase by one. In theory, since strings are immutable, an implementation could re-use memory for substring slices. This would likely complicate any reference-counting based garbage collection process, so it may not be a useful idea in practice. Consider the case where a small slice from a much larger string was taken - unless you implemented some kind of sub-reference counting on the slice, the memory from the much larger string could not be freed until the end of the substring's lifetime. For users that specifically need a standard type which can be sliced without copying the underlying data, there is memoryview. See What exactly is the point of memoryview in Python for more information about that.