eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using multiple notebooks in PySpark and import variables across these notebooks using %run path. Every time I run the command, all variables that I displayed in the original notebook are being displayed again in the current notebook (the notebook in which I %run). But I do not want them to be displayed in the current notebook. I only want to be able to work with the imported variables. How do I suppress the output being display every time? Note, I am not sure if it matters, but I am working in DataBricks. Thank you! Command example: %run /Users/myemail/Nodebook 

Answer: You can use the "Hide Result" option in the upper right toggle of the cell:
Question: I want to test a Python function that reads a gzip file and extracts something from the file (using pytest). import gzip def my_function(file_path): output = [] with gzip.open(file_path, 'rt') as f: for line in f: output.append('something from line') return output  Can I create a gzip file like object that I can pass to my_function? The object should have defined content and should work with gzip.open() I know that I can create a temporary gzip file in a fixture but this depends on the filesystem and other properties of the environment. Creating a file-like object from code would be more portable.

Answer: You can use the io and gzip libraries to create in-memory file objects. Example: import io, gzip def inmem(): stream = io.BytesIO() with gzip.open(stream, 'wb') as f: f.write(b'spam\neggs\n') stream.seek(0) return stream
Question: This question might seem like a duplicate of this. I am trying to run Apache Beam python pipeline using flink on an offline instance of Kubernetes. However, since I have user code with external dependencies, I am using the Python SDK harness as an External Service - which is causing errors (described below). The kubernetes manifest I use to launch the beam python SDK: apiVersion: apps/v1 kind: Deployment metadata: name: beam-sdk spec: replicas: 1 selector: matchLabels: app: beam component: python-beam-sdk template: metadata: labels: app: beam component: python-beam-sdk spec: hostNetwork: True containers: - name: python-beam-sdk image: apachebeam/python3.7_sdk:latest imagePullPolicy: "Never" command: ["/opt/apache/beam/boot", "--worker_pool"] ports: - containerPort: 50000 name: yay apiVersion: v1 kind: Service metadata: name: beam-python-service spec: type: NodePort ports: - name: yay port: 50000 targetPort: 50000 selector: app: beam component: python-beam-sdk  When I launch my pipeline with the following options: beam_options = PipelineOptions([ "--runner=FlinkRunner", "--flink_version=1.9", "--flink_master=10.101.28.28:8081", "--environment_type=EXTERNAL", "--environment_config=10.97.176.105:50000", "--setup_file=./setup.py" ])  I get the following error message (within the python sdk service): NAME READY STATUS RESTARTS AGE beam-sdk-666779599c-w65g5 1/1 Running 1 4d20h flink-jobmanager-74d444cccf-m4g8k 1/1 Running 1 4d20h flink-taskmanager-5487cc9bc9-fsbts 1/1 Running 2 4d20h flink-taskmanager-5487cc9bc9-zmnv7 1/1 Running 2 4d20h (base) [~]$ sudo kubectl logs -f beam-sdk-666779599c-w65g5 2020/02/26 07:56:44 Starting worker pool 1: python -m apache_beam.runners.worker.worker_pool_main --service_port=50000 --container_executable=/opt/apache/beam/boot Starting worker with command ['/opt/apache/beam/boot', '--id=1-1', '--logging_endpoint=localhost:39283', '--artifact_endpoint=localhost:41533', '--provision_endpoint=localhost:42233', '--control_endpoint=localhost:44977'] 2020/02/26 09:09:07 Initializing python harness: /opt/apache/beam/boot --id=1-1 --logging_endpoint=localhost:39283 --artifact_endpoint=localhost:41533 --provision_endpoint=localhost:42233 --control_endpoint=localhost:44977 2020/02/26 09:11:07 Failed to obtain provisioning information: failed to dial server at localhost:42233 caused by: context deadline exceeded  I have no idea what the logging- or artifact endpoint (etc.) is. And by inspecting the source code it seems like that the endpoints has been hard-coded to be located at localhost.

Answer: (You said in a comment that the answer to the referenced post is valid, so I'll just address the specific error you ran into in case someone else hits it.) Your understanding is correct; the logging, artifact, etc. endpoints are essentially hardcoded to use localhost. These endpoints are meant to be only used internally by Beam and are not configurable. So the Beam worker is implicitly assumed to be on the same host as the Flink task manager. Typically, this is accomplished by making the Beam worker pool a sidecar of the Flink task manager pod, rather than a separate service.
Question: I am using matplotlib.pyplot module imported as plt for plots.  In the plt.plot() statement, if I pass the arguments as "x= array1, "y= array2", I get "TypeError: plot got an unexpected keyword argument 'x' ".  The code gets executed correctly if I simple pass "array1 and array2", without explicitly saying they correspond to x and y axes.  Why is that?

Answer: If you look at the function definition, https://github.com/matplotlib/matplotlib/blob/9a24fb724331f50baf0da4d17188860357d328a9/lib/matplotlib/axes/_axes.py#L72, you can see the asterisk there, and the use of that doesn't work with using keywords for non-optional parameters. See Python args and kwargs: Demystified for example. def plot(self, *args, scalex=True, scaley=True, data=None, **kwargs): """ Plot y versus x as lines and/or markers. Call signatures:: plot([x], y, [fmt], *, data=None, **kwargs) plot([x], y, [fmt], [x2], y2, [fmt2], ..., **kwargs)
Question: I am running a dockerized django app and I am looking for a way to run (a) directive(s) every time before I build a docker container. More concretely, I would like to run docker-compose -f production.yml run --rm django python manage.py check --deploy each time before I either build or up the production.yml file and stop the build process if any erroroccur. Like a pre-hook.  I know I could achieve this with a bash-script, yet I was wondering if there is a way of doing this inside the docker-compose file. I can't find anything in the docker documentation (except events, but I don't understand if they serve for what I want to achieve) about it and I assume that this is not possible. Yet, maybe it is in fact possible or maybe there is a hacky workaround?  Thanks in advance for any tips. 

Answer: Currently, this is not possible. There have been multiple requests to add such functionality, but the maintainers do not consider this a good idea. See:  https://github.com/docker/compose/issues/468 https://github.com/docker/compose/issues/1341 https://github.com/docker/compose/issues/6736
Question: Using python, the method WebDriverWait is used to wait for 1 element to be present on the webpage. How can this method be used without multiple try/except? Is there an OR option for multiple cases using this method? https://selenium-python.readthedocs.io/waits.html

Answer: Without using multiple try/except{} to induce WebDriverWait for two elements through OR option you can use either of the following solutions:  Using CSS_SELECTOR: element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".element_A_class, .element_B_class"))   Using XPATH through lambda: element = WebDriverWait(driver,20).until(lambda driver: driver.find_element(By.XPATH,"element_A_xpath") or driver.find_element(By.XPATH,"element_B_xpath"))     Reference You can find a couple of relevant discussions in:  selenium two xpath tests in one Python / Selenium: Logic Operators in WebDriverWait Expected Conditions
Question: I keep getting this error when I start Rabbitmq and wonder what's wrong?  BOOT FAILED =========== Error description: init:do_boot/3 line 817 init:start_em/1 line 1109 rabbit:start_it/1 line 474 rabbit:broker_start/1 line 350 rabbit:start_loaded_apps/2 line 600 app_utils:manage_applications/6 line 126 lists:foldl/3 line 1263 rabbit:'-handle_app_error/1-fun-0-'/3 line 723 throw:{could_not_start,ra, {ra, {{shutdown, {failed_to_start_child,ra_system_sup, {shutdown, {failed_to_start_child,ra_log_sup, {shutdown, {failed_to_start_child,ra_log_wal_sup, {shutdown, {failed_to_start_child,ra_log_wal, {{case_clause,{ok,<<0,0,0,0,0>>}}, [{ra_log_wal,open_existing,1, [{file,"src/ra_log_wal.erl"},{line,646}]}, {ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1, [{file,"src/ra_log_wal.erl"},{line,265}]}, {ra_log_wal,recover_wal,2, [{file,"src/ra_log_wal.erl"},{line,268}]}, {ra_log_wal,init,1, [{file,"src/ra_log_wal.erl"},{line,214}]}, {gen_batch_server,init_it,6, [{file,"src/gen_batch_server.erl"},{line,133}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}}, {ra_app,start,[normal,[]]}}}} Log file(s) (may contain more information): C:/Users/AIMLExpert/AppData/Roaming/RabbitMQ/log/rabbit@DESKTOP-N0Q3S7C.log C:/Users/AIMLExpert/AppData/Roaming/RabbitMQ/log/rabbit@DESKTOP-N0Q3S7C_upgrade.log {"init terminating in do_boot",{could_not_start,ra,{ra,{{shutdown,{failed_to_start_child,ra_system_sup,{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,{{case_clause,{ok,<<0,0,0,0,0>>}},[{ra_log_wal,open_existing,1,[{file,"src/ra_log_wal.erl"},{line,646}]},{ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1,[{file,"src/ra_log_wal.erl"},{line,265}]},{ra_log_wal,recover_wal,2,[{file,"src/ra_log_wal.erl"},{line,268}]},{ra_log_wal,init,1,[{file,"src/ra_log_wal.erl"},{line,214}]},{gen_batch_server,init_it,6,[{file,"src/gen_batch_server.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}},{ra_app,start,[normal,[]]}}}}} init terminating in do_boot ({could_not_start,ra,{ra,{{shutdown,{_}},{ra_app,start,[_]}}}}) Crash dump is being written to: C:\Users\AIMLExpert\AppData\Roaming\RabbitMQ\log\erl_crash.dump...done  whats causing this Error? I have tried restarting the command line but am still getting the error.

Answer: I had the same issue on Arch Linux due to a crash dump. It was saying in the logs that it had problem with reverting a WAL file that was 0 bytes in size. After removing the WAL file the service started. Locate the WAL in /var/lib/rabbitmq/mnesia with find /var/lib/rabbitmq/ -name "*.wal" and remove it. Restart the service afterwards.
Question: I have been trying to capture WIFI packets with Linux and see the frequency/channel at which packet was captured. I tried Wireshark and there was no luck and no help. Though using a sample packets from Wireshark, I can see the frequency/channel. So now I'm experimenting with Scapy. I wanted to figure out the frequency/channel of a sniffed packet, but still no luck. Is there a way to do this with Scapy. P.S. If there is a better tool than Scapy, or Python, I appreciate comments

Answer: I found out that RadioTab headers are not part of any Dot11 protocol but are merely added by the network interface. And the reason I got the RadioTab headers on sample packets from Wireshark.org and not from my live wireshark capture is because some network adapters do not add RadioTap while others do and the network adapter of my laptop does not add RadioTab headers. I checked this with a new external WiFi adapter and it did add the RadioTap headers.  If the adapter does not inject the additional information as it captures frames, then no radiotap headers will be added.  So to my main question, how to get/set frequency of a packet. I expected Scapy to have this option but it doesn't, and it shouldn't. The reason is that the frequency depends on what is set on the network adapter. So what I did was to set the frequency/channel of my WiFi adapter to a different one. My external WiFi adapter can work in various channels so I changed each and confirmed with the RadioTap header. There are a simple linux commands/tools that helped me check the supported channels of my WiFi interface, and switch to a particular channel.  To capture/send packets at a certain frequency or channel, you need to change the working channel of your interface and set the sniffer/sender interface in scapy to that interface.  EDIT - Other problems I faced and solutions: If you are on linux, and you want to change the working channel of your interface you need to disable network-manager for that interface and to do this First Add the following snippet to /etc/network/interfaces auto $iface iface $iface inet dhcp wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf  replace $iface with your interface name. This will let you control the interface by yourself. And then add the following lines to /etc/wpa_supplicant/wpa_supplicant.conf ctrl_interface=/var/run/wpa_supplicant network={ ssid="Your_AP_SSID" psk="Your_Passphrase" freq_list=2412 2437 2462 }  Note that 2412 2437 2462 are the frequencies (channel 1, 6, 11 in this case) for your interface to choose from. You can edit them to desired frequency. Source. But first you have to check that your interface supports these frequencies. To check that iwlist channel  Finally after everything is done.  sendp(Ether()/IP(dst="1.2.3.4",ttl=(1,4)), iface="wlp3s0")  This will send you packets at the frequency that wlp3s0 is set.
Question: Hi I am trying to debug multi processes in python. This below is a portion of where I run multi-processes using Pool pool = Pool(num_half_logical_cpus) pool_result_dict = pool.starmap(process_batches, lstListSets)  However, I can't hit any breakpoints. Can anyone guide me to hit those breakpoints I set up? Thanks!

Answer: Add this option to launch.json will let you debug multiple processes. "subProcess": true,  Then it will hit the breakpoint then you are able to select which process you want to step through (F10). There will be a list of processes at the lower-left sub-window.
Question: I am learning how to use Azure functions and using my web scraping script in it.  It uses BeautifulSoup (bs4) and pymysql modules. It works fine when I tried it locally in the virtual environment as per this MS guide: https://learn.microsoft.com/en-us/azure/azure-functions/functions-create-first-azure-function-azure-cli?pivots=programming-language-python&tabs=cmd%2Cbrowser#run-the-function-locally But when I create the function App and publish the script to it, Azure Functions logs give me this error: Failure Exception: ModuleNotFoundError: No module named 'pymysql'.  It must happen when attempting to import it. I really don't know how to proceed, where should I specify what modules it needs to install?

Answer: You need to check if you have generated the requirements.txt which includes all of the information of the modules. When you deploy the function to azure, it will install the modules by the requirements.txt automatically. You can generate the information of modules in requirements.txt file by the command below in local: pip freeze > requirements.txt  And then deploy the function to azure by running the publish command: func azure functionapp publish hurypyfunapp --build remote  For more information about deploy python function from local to auzre, please refer to this tutorial. By the way, if you use consumption plan for your python function, the "Kudu" is not available for us. If you want to use "Kudu", you need to create app service plan for it but not consumption plan. Hope it helps~
Question: Please help me understand why the following doesn't work. In particular - instance attributes of a tested class are not visible to Python's unittest.Mock. In the example below bar instance attribute is not accessible. The error returned is: AttributeError: <class 'temp.Foo'> does not have the attribute 'bar'  import unittest from unittest.mock import patch class Foo: def __init__(self): super().__init__(self) self.bar = some_external_function_returning_list() def do_someting(self): calculate(self.bar) class TestFoo(unittest.TestCase): @patch('temp.Foo.bar') def test_do_something(self, patched_bar): patched_bar.return_value = ['list_elem1', 'list_elem2'] 

Answer: Patching is used to modify name or attribute lookup. In this case, there is no bar attribute of the class temp.Foo. If the intent is to patch the instance variable, you either need an existing instance to modify def test(self): f = Foo() with patch.object(f, 'bar', 3): self.assertEqual(f.bar, 3)  or you may want to patch the function call that initializes the instance attribute in the first place. def test(self): with patch('some_external_function_returning_list', return_value=3): f = Foo() self.assertEqual(f.bar, 3)
Question: Let's say I have a Python list representing ranges for some variables: conditions = [['i', (1, 5)], ['j', (1, 2)]]  This represents that variable i ranges from 1 to 5, and inside that loop variable j ranges from 1 to 2. I want a dictionary for each possible combination: {'i': 1, 'j': 1} {'i': 1, 'j': 2} {'i': 2, 'j': 1} {'i': 2, 'j': 2} {'i': 3, 'j': 1} {'i': 3, 'j': 2} {'i': 4, 'j': 1} {'i': 4, 'j': 2} {'i': 5, 'j': 1} {'i': 5, 'j': 2}  The reason is that I want to iterate over them. But because the whole space is too big, I do not want to generate all of them, store them and then iterate over that list of dictionaries. I thought about using the following recursive procedure, but I need some help with the yield part. Where should it be? How do I avoid nested generators? def iteration(conditions, currentCondition, valuedIndices): if currentCondition == len(conditions): yield valuedIndices else: cond = conditions[currentCondition] index = cond[0] lim1 = cond[1][0] lim2 = cond[1][1] for ix in range(lim1, lim2 + 1): valuedIndices[index] = ix yield iteration(conditions, currentCondition + 1, valuedIndices)  Now I would like to be able to do: for valued_indices in iteration(conditions, 0, {}): ... 

Answer: This is a case where it might be easier to take a step back and start fresh. Let's start by getting the keys and the intervals separate, using a well-known trick involving zip: >>> keys, intervals = list(zip(*conditions)) >>> keys ('i', 'j') >>> intervals ((1, 5), (1, 2))  (The correspondence between the two preserves the original pairing; intervals[i] is the interval for the variable keys[i] for all i.) Now, let's create proper range objects from those intervals >>> intervals = [range(x, y+1) for x, y in intervals] >>> list(intervals[0]) [1, 2, 3, 4, 5] >>> list(intervals[1]) [1, 2]  We can compute the product of these range objects >>> for v in product(*intervals): ... print(v) ... (1, 1) (1, 2) (2, 1) (2, 2) (3, 1) (3, 2) (4, 1) (4, 2) (5, 1) (5, 2)  which you should recognize as the values to use for each dict. You can zip each of those values with the keys to create an appropriate set of arguments for the dict command. For example: >>> dict(zip(keys, (1,1))) {'i': 1, 'j': 1}  Putting this all together, we can iterate over the product to produce each dict in turn, yielding it. def iteration(conditions): keys, intervals = zip(*conditions) intervals = [range(x, y+1) for x, y in intervals] yield from (dict(zip(keys, v)) for v in product(*intervals))
Question: I encountered a strange behavior of np.ndarray.tobytes() that makes me doubt that it is working deterministically, at least for arrays of dtype=object. import numpy as np print(np.array([1,[2]]).dtype) # => object print(np.array([1,[2]]).tobytes()) # => b'0h\xa3\t\x01\x00\x00\x00H{!-\x01\x00\x00\x00' print(np.array([1,[2]]).tobytes()) # => b'0h\xa3\t\x01\x00\x00\x00\x88\x9d)-\x01\x00\x00\x00'  In the sample code, a list of mixed python objects ([1, [2]]) is first converted to a numpy array, and then transformed to a byte sequence using tobytes().  Why do the resulting byte-representations differ for repeated instantiations of the same data? The documentation just states that it converts an ndarray to raw python bytes, but it does not refer to any limitations. So far, I observed this just for dtype=object. Numeric arrays always yield the same byte sequence: np.random.seed(42); print(np.random.rand(3).tobytes()) # b'\xecQ_\x1ew\xf8\xd7?T\xd6\xbbh@l\xee?Qg\x1e\x8f~l\xe7?' np.random.seed(42); print(np.random.rand(3).tobytes()) # b'\xecQ_\x1ew\xf8\xd7?T\xd6\xbbh@l\xee?Qg\x1e\x8f~l\xe7?'  Have I missed an elementar thing about python's/numpy's memory architecture? I tested with numpy version 1.17.2 on a Mac.  Context: I encountered this problem when trying to compute a hash for arbitrary data structures. I hoped that I can rely on the basic serialization capabilities of tobytes(), but this appears to be a wrong premise. I know that pickle is the standard for serialization in python, but since I don't require portability and my data structures only contain numbers, I first sought help with numpy.

Answer: An array of dtype object stores pointers to the objects it contains. In CPython, this corresponds to the id. Every time you create a new list, it will be allocated at a new memory address. However, small integers are interned, so 1 will reference the same integer object every time. You can see exactly how this works by checking the IDs of some sample objects: >>> x = np.array([1, [2]]) >>> x.tobytes() b'\x90\x91\x04a\xfb\x7f\x00\x00\xc8[J\xaa+\x02\x00\x00' >>> id(x[0]) 140717641208208 >>> id(1) # Small integers are interned 140717641208208 >>> id(x[0]).to_bytes(8, 'little') # Checks out as the first 8 bytes b'\x90\x91\x04a\xfb\x7f\x00\x00' >>> id(x[1]).to_bytes(8, 'little') # Checks out as the last 8 bytes b'\xc8[J\xaa+\x02\x00\x00'  As you can see, it is quite deterministic, but serializes information that is essentially useless to you. The operation is the same for numeric arrays as for object arrays: it returns a view or copy of the underlying buffer. The contents of the buffer is what is throwing you off. Since you mentioned that you are computing hashes, keep in mind that there is a reason that python lists are unhashable. You can have lists that are equal at one time and different at another. Using IDs is generally not a good idea for an effective hash.
Question: Firstly, I'm working on an Amazon EC2 instance, Amazon linux version 2 AMI using Python 3.7. I'm trying to install the python-Levenshtein package using the command: pip3 install python-Levenshtein --user  and I'm getting a rather huge error, with the key parts being; gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c Levenshtein/_levenshtein.c -o build/temp.linux-x86_64-3.7/Levenshtein/_levenshtein.o Levenshtein/_levenshtein.c:99:10: fatal error: Python.h: No such file or directory #include <Python.h> ^~~~~~~~~~ compilation terminated. error: command 'gcc' failed with exit status 1  and: Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-m87wdfsg/python-Levenshtein/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-w3meudfd-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-m87wdfsg/python-Levenshtein/  I've tried many, many solutions, the main one being this: error: command 'gcc' failed with exit status 1 while installing eventlet Edit Tried this... 1) sudo yum install python-devel (no change) 2) reinstalling GCC (no change) 3) spinning up a clean EC2 instance (same error) 4) pip install python-dev-tools (now gives me the error repeated 2x) 5) attempting to find Python.h using 'locate "Python.h"' (nothing) 6) sudo yum list python37-devel (error: no matching packages) 

Answer: Thanks to Charles, the answer is as follows:  sudo yum install python3-devel