eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm setting up virtual env. I was getting warnings about an outdated pip (19.2) so I updated pip on my (macos) system globally, sudo -H python3 -m pip install --upgrade pip. It seems to have worked, but when I make a new venv, I'm still getting the old pip version. % pip --version pip 20.1 from /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip (python 3.8) % python3 -m pip --version pip 20.1 from /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip (python 3.8) % rm -rf .venv # make sure % python3 -m venv .venv % . .venv/bin/activate (.venv) % python3 -m pip --version pip 19.2.3 from /Users/marvin/.venv/lib/python3.8/site-packages/pip (python 3.8) (.venv) % pip --version pip 19.2.3 from /Users/marvin/.venv/lib/python3.8/site-packages/pip (python 3.8)  Where is the older version coming from?

Answer: Pip is installed anew in any freshly created venv. The venv's default pip version is associated with the Python version, and is completely independent from whatever pip version you may have installed on the system. The older version comes from a wheel file bundled with the stdlib ensurepip module. This allows users to create a venv even with no internet connection available, as the venv docs mention:  Unless the --without-pip option is given, ensurepip will be invoked to bootstrap pip into the virtual environment  You can check the bundled pip version with ensurepip.version: >>> import ensurepip >>> ensurepip.version() '19.2.3'  Python 3.8.2 is vendoring pip 19.2.3 and setuptools 41.2.0, matching what you've seen. To create venvs directly with the latest pip version, rather than creating them with an older pip and then upgrading the pip version, refer to this answer: How to get “python -m venv” to directly install latest pip version
Question: I have a JSON file. I am running a program, in python, where data is extracted from the JSON file. Is there any way to encrypt the JSON file with a key, so that if someone randomly opens the file, it would be a mess of characters, but when the key is fed to the program, it decrypts it and is able to read it? Thanks in advance.

Answer: Yes, you can encrypt a .json file. Make sure you install the cryptography package by typing pip install cryptography # or on windows: python -m pip install cryptography  Then, you can make a program similar to mine: #this imports the cryptography package from cryptography.fernet import Fernet #this generates a key and opens a file 'key.key' and writes the key there key = Fernet.generate_key() with open('key.key','wb') as file: file.write(key) #this just opens your 'key.key' and assings the key stored there as 'key' with open('key.key','rb') as file: key = file.read() #this opens your json and reads its data into a new variable called 'data' with open('filename.json','rb') as f: data = f.read() #this encrypts the data read from your json and stores it in 'encrypted' fernet = Fernet(key) encrypted = fernet.encrypt(data) #this writes your new, encrypted data into a new JSON file with open('filename.json','wb') as f: f.write(encrypted)  Note that this block: with open('key.key','wb') as file: file.write(key) #this just opens your 'key.key' and assigns the key stored there as 'key' with open('key.key','rb') as file: key = file.read()  isn't necessary. It is just a way to store the generated key in a safe place, and read it back in. You can delete that block if you want. Let me know if you need further assistance :)
Question: Suppose my main.py is like this (this is a simplified example, in my app I use an actual database and I have two different database URIs for development and testing): from fastapi import FastAPI from pydantic import BaseSettings app = FastAPI() class Settings(BaseSettings): ENVIRONMENT: str class Config: env_file = ".env" case_sensitive = True settings = Settings() databases = { "dev": "Development", "test": "Testing" } database = databases[settings.ENVIRONMENT] @app.get("/") def read_root(): return {"Environment": database}  while the .env is ENVIRONMENT=dev  Suppose I want to test my code and I want to set ENVIRONMENT=test to use a testing database. What should I do? In FastAPI documentation (https://fastapi.tiangolo.com/advanced/settings/#settings-and-testing) there is a good example but it is about dependencies, so it is a different case as far as I know. My idea was the following (test.py): import pytest from fastapi.testclient import TestClient from main import app @pytest.fixture(scope="session", autouse=True) def test_config(monkeypatch): monkeypatch.setenv("ENVIRONMENT", "test") @pytest.fixture(scope="session") def client(): return TestClient(app) def test_root(client): response = client.get("/") assert response.status_code == 200 assert response.json() == {"Environment": "Testing"}  but it doesn't work. Furthermore I get this error: ScopeMismatch: You tried to access the 'function' scoped fixture 'monkeypatch' with a 'session' scoped request object, involved factories test.py:7: def test_config(monkeypatch) env\lib\site-packages\_pytest\monkeypatch.py:16: def monkeypatch()  while from pytest official documentation it should work (https://docs.pytest.org/en/3.0.1/monkeypatch.html#example-setting-an-environment-variable-for-the-test-session). I have the latest version of pytest installed. I tried to use specific test environment variables because of this: https://pydantic-docs.helpmanual.io/usage/settings/#field-value-priority. To be honest I'm lost, my only real aim is to have a different test configuration (in the same way Flask works: https://flask.palletsprojects.com/en/1.1.x/tutorial/tests/#setup-and-fixtures). Am I approaching the problem the wrong way?

Answer: PydanticSettings are mutable, so you can simply override them in your test.py: from main import settings settings.ENVIRONMENT = 'test'
Question: I need to apply an exponential decay of learning rate every 10 epochs. Initial learning rate is 0.000001, and decay factor is 0.95 is this the proper way to set it up? lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay( initial_learning_rate=0.000001, decay_steps=(my_steps_per_epoch*10), decay_rate=0.05) opt = tf.keras.optimizers.SGD(learning_rate=lr_schedule, momentum=0.9)  The formula of exponential decay is current_lr = initial_lr * (1 - decay_factor)^t Except that in the code it is implemented as : decayed_learning_rate = learning_rate * decay_rate ^ (global_step / decay_steps)  To my knowledge, decay_rate should be 1 - decay_factor and decay_steps should mean how many steps are performed before applying the decay, in my case my_steps_per_epoch*10. Is that correct? EDIT: If I pause and save my model (using callbacks) after the 10th epoch, and then resume by loading the model and calling model.fit with initial_epoch=10 and epochs=11, will it start in the 11th epoch and apply the exponential decay?

Answer: decay_steps can be used to state after how many steps (processed batches) you will decay the learning rate. I find it quite useful to just specify the initial and the final learning rate and calculate the decay_factor automatically via the following: initial_learning_rate = 0.1 final_learning_rate = 0.0001 learning_rate_decay_factor = (final_learning_rate / initial_learning_rate)**(1/epochs) steps_per_epoch = int(train_size/batch_size) lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay( initial_learning_rate=initial_learning_rate, decay_steps=steps_per_epoch, decay_rate=learning_rate_decay_factor, staircase=True)
Question: After building a flutter web app with  flutter build web  I want to serve the app using a simple Django server. How can I do that?

Answer: After running the command, copy the build\web directory from the flutter project to your django project. Let's say you put it at the root and renamed it to flutter_web_app. So we have, djangoproject | djangoproject | settings.py | urls.py | ... | flutter_web_app | ... | other apps  Now edit the base tag in flutter_web_app/index.html to give the application a prefix. Note that the prefix/base should start and end with a /. <head> ... <base href="/flutter_web_app/"> ... </head>  Now, in your djangoproject/urls.py, match the prefix and serve your flutter application. from django.urls import path from django.views.static import serve import os BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) FLUTTER_WEB_APP = os.path.join(BASE_DIR, 'flutter_web_app') def flutter_redirect(request, resource): return serve(request, resource, FLUTTER_WEB_APP) urlpatterns = [ ... path('flutter_web_app/', lambda r: flutter_redirect(r, 'index.html')), path('flutter_web_app/<path:resource>', flutter_redirect), ]  Running this locally, url 127.0.0.1/flutter_web_app will serve our flutter_web_app/index.html. All the files required by the flutter application are prefixed with flutter_web_app. For eg. main.dart.js is requested with a GET to flutter_web_app/main.dart.js. We extract the path after the prefix as resource and serve the corresponding file from flutter_web_app directory (flutter_redirect function).
Question: I have a table for which when processing records I get either the full record, else only the columns to be updated.  I want to write a query to handle updates but only update the columns with non null values. For example, Existing table: 1 | John Doe | USA 2 | Jane Doe | UK  Incoming records: (3, Kate Bill, Canada) (2, null, USA)  I want to insert the first record and on conflict of key on second record ONLY update the last column.  I'm not sure how to write this using a execute_values method call: execute_values(cursor, "INSERT INTO user_data\ (id, name, country) VALUES %s ON CONFLICT DO UPDATE SET \ <how to only set non null values here>", vendor_records)  I'm using psycopg2 to execute this.

Answer: In the on conflict (pay particular attention to the do update set name = expression) you can use coalesce with the excluded values to update columns to the specified values (if not null), or the existing values (if specified is null); This would be that format: -- setup create table test1(id integer primary key, col1 text,col2 text); insert into test1(id, col1, col2) values (1, 'John Doe', 'USA') , (2, 'Jane Doe', 'UK'); select * from test1; -- test on conflict insert into test1 as t(id, col1, col2) values (3, 'Kate Bill', 'Canada') , (2, null, 'USA') on conflict(id) do update set col1 = coalesce(excluded.col1, t.col1) , col2 = coalesce(excluded.col2, t.col2); -- validate select * from test1;
Question: I'm following along a course - Django development to deployment. After pulling it to Digital Ocean everything else ran smoothly. Until  I tried running python manage.py help (env) djangoadmin@ubuntu-1:~/pyapps/btre_project_4$ python manage.py help  and I get this error. Traceback (most recent call last): File "manage.py", line 21, in <module> main() File "manage.py", line 17, in main execute_from_command_line(sys.argv) File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line utility.execute() File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/core/management/__init__.py", line 377, in execute django.setup() File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/__init__.py", line 16, in setup from django.urls import set_script_prefix File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/urls/__init__.py", line 1, in <module> from .base import ( File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/urls/base.py", line 9, in <module> from .exceptions import NoReverseMatch, Resolver404 File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/urls/exceptions.py", line 1, in <module> from django.http import Http404 File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/http/__init__.py", line 2, in <module> from django.http.request import ( File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/http/request.py", line 10, in <module> from django.core import signing File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/core/signing.py", line 45, in <module> from django.utils.crypto import constant_time_compare, salted_hmac File "/home/djangoadmin/pyapps/env/lib/python3.5/site-packages/django/utils/crypto.py", line 6, in <module> import secrets ImportError: No module named 'secrets'  I'm a newbie and have been stuck on this for a while. I just want to know what could possibly cause this.

Answer: The secrets module was added to Python in version 3.6. Your host is using Python 3.5, hence the secrets module is unavailable. You need a host with Python 3.6+, or a version of Django that doesn't depend on the secrets module
Question: When using the code below the sound plays: import IPython.display as ipd import numpy sr = 22050 # sample rate T = 0.5 # seconds t = numpy.linspace(0, T, int(T*sr), endpoint=False) # time variable x = 0.5*numpy.sin(2*numpy.pi*440*t) # pure sine wave at 440 Hz ipd.Audio(x, rate=sr, autoplay=True) # load a NumPy array  But when I use it inside a function it stops working: import IPython.display as ipd import numpy def SoundNotification(): sr = 22050 # sample rate T = 0.5 # seconds t = numpy.linspace(0, T, int(T*sr), endpoint=False) # time variable x = 0.5*numpy.sin(2*numpy.pi*440*t) # pure sine wave at 440 Hz ipd.Audio(x, rate=sr, autoplay=True) # load a NumPy array SoundNotification()  I've tried to assign the audio to a variable and return it which works: import IPython.display as ipd import numpy def SoundNotification(): sr = 22050 # sample rate T = 0.5 # seconds t = numpy.linspace(0, T, int(T*sr), endpoint=False) # time variable x = 0.5*numpy.sin(2*numpy.pi*440*t) # pure sine wave at 440 Hz sound = ipd.Audio(x, rate=sr, autoplay=True) # load a NumPy array return sound sound = SoundNotification() sound  But I want to use the sound in a different function: import IPython.display as ipd import numpy def SoundNotification(): sr = 22050 # sample rate T = 0.5 # seconds t = numpy.linspace(0, T, int(T*sr), endpoint=False) # time variable x = 0.5*numpy.sin(2*numpy.pi*440*t) # pure sine wave at 440 Hz sound = ipd.Audio(x, rate=sr, autoplay=True) # load a NumPy array return sound def WhereIWantToUseTheSound(): sound = SoundNotification() sound WhereIWantToUseTheSound()  How do I make this work and what causes this behavior? The kernel for the notebook is Python 3. Edit: I want to play the sound in a scheduled event: import IPython.display as ipd import numpy import sched, time sound = [] def SoundNotification(): sr = 22050 # sample rate T = 0.5 # seconds t = numpy.linspace(0, T, int(T*sr), endpoint=False) # time variable x = 0.5*numpy.sin(2*numpy.pi*440*t) # pure sine wave at 440 Hz sound = ipd.Audio(x, rate=sr, autoplay=True) # load a NumPy array return sound def do_something(sc): print("Doing stuff...") # do your stuff sound_ = SoundNotification() s.enter(interval, 1, do_something, (sc,)) return sound_ s = sched.scheduler(time.time, time.sleep) interval = int(input("Interval between captures in seconds: ")) s.enter(0, 1, do_something, (s,)) s.run()  I don't know how to return the sound and schedule the next event within the same function.

Answer: I was having this same problem, the sound was played when I called: from IPython.display import Audio Audio('/path/beep.mp3', autoplay=True)  But it didn't work when it was inside a function. The problem is that the function call doesn't really play the sound, it's actually played by the resulting HTML that is returned to Jupyter output. So to overcome this, you can force the function to render the HTML using display( ) function from IPython. This will work: from IPython.display import Audio from IPython.core.display import display def beep(): display(Audio('/path/beep.mp3', autoplay=True)) beep();
Question: I am using FastAPI to write a web service. It is good and fast. FastAPI is using pydantic models to validate input and output data, everything is good but when I want to declare a nested model for array of jsons like below: [ { "name": "name1", "family": "family1" }, { "name": "name2", "family": "family2" } ]  I get empty response. I think there is a problem with my model which is: class Test(BaseModel): name: str family: str class Config: orm_mode = True class Tests(BaseModel): List[Test] class Config: orm_mode = True  So, my question is how should I write a model for array of jsons?

Answer: Update (26/09/2020) In Python 3.9 (not yet released), you can do the same as below but with the built-in list generic type (which is always in scope) rather than needing to import the capitalized List type from typing, e.g. @app.get("/tests", response_model=list[Test])   The issue here is that you are trying to create a pydantic model where it is not needed. If you want to serialize/deserialize a list of objects, just wrap your singular model in a List[] from python's builtin typing module. There is no need to try to create a plural version of your object with a pydantic BaseModel (and as you can see, it does not work anyway). With that said, the simplest way to do what you want is to just specify a List[Test] at any point where you need a list of Tests, e.g. from typing import List from fastapi import FastAPI from pydantic import BaseModel existing_tests = [ { "name": "name1", "family": "family1" }, { "name": "name2", "family": "family2" } ] class Test(BaseModel): name: str family: str class Config: orm_mode = True app = FastAPI() @app.get("/tests", response_model=List[Test]) async def fetch_tests(): return existing_tests @app.post("/tests") async def submit_tests(new_tests: List[Test]): print(new_tests)  But of course if you find yourself repeatedly (or only) specifying Test as a list, you can of course just assign this to a variable and then use that variable where needed, like so: Tests = List[Test] @app.get("/tests", response_model=Tests) async def fetch_tests(): return existing_tests @app.post("/tests") async def submit_tests(new_tests: Tests): print(new_tests)  I think the first option is probably slightly clearer in your code though, and unless you are specifying List[Test] many times, using a variable for this purpose is probably not worth the extra layer of indirection.
Question: I have updated Selenium but the error keeps occurring even though the web page loads. However, in some instances, the driver starts but it is stagnant. Is this causing an issue and if so, how do I resolve it? [11556:9032:0502/152954.314:ERROR:device_event_log_impl.cc(162)] [15:29:54.314] Bluetooth: bluetooth_adapter_winrt.cc:1055 Getting Default Adapter failed. 

Answer: This error message... ERROR:device_event_log_impl.cc(162)] [15:29:54.314] Bluetooth: bluetooth_adapter_winrt.cc:1055 Getting Default Adapter failed.  ...implies that ScopedClosureRunner on_init failed in BluetoothAdapterWinrt::OnGetDefaultAdapter().  Analysis This error is defined in bluetooth_adapter_winrt.cc as follows: void BluetoothAdapterWinrt::OnGetDefaultAdapter( base::ScopedClosureRunner on_init, ComPtr<IBluetoothAdapter> adapter) { DCHECK_CALLED_ON_VALID_THREAD(thread_checker_); if (!adapter) { BLUETOOTH_LOG(ERROR) << "Getting Default Adapter failed."; return; }   Solution Ensure that:  Selenium is upgraded to current levels Version 3.141.59. ChromeDriver is updated to current ChromeDriver v84.0 level. Chrome is updated to current Chrome Version 84.0 level. (as per ChromeDriver v84.0 release notes) If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.   Additional considerations However it was observed that this error can be supressed by running Chrome as root user (administrator) on Linux. but that would be a deviation from the documentation in ChromeDriver - WebDriver for Chrome where it is mentioned:  A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing '--no-sandbox' flag when creating your WebDriver session, i.e. the ChromeDriver session as such a configuration is unsupported and highly discouraged.  Ideally, you need to configure your environment to run Chrome as a regular user instead.  Suppressing the error Finally, as per the documentation in Selenium Chrome Driver: Resolve Error Messages Regarding Registry Keys and Experimental Options these error logs can be supressed by adding the argument: excludeSwitches: ['enable-logging']  So your effective code block will be: from selenium import webdriver options = webdriver.ChromeOptions() options.add_experimental_option("excludeSwitches", ["enable-logging"]) driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe') driver.get("https://www.google.com/")
Question: I can automate Outlook on windows with win32/COM, but does anyone know of a pure python way to do the same on mac osx? A simple use case would be:  Open outlook/connect to the active instance Launch a blank new email  I want to create an app to create email templates and attach files, then let the user finish editing the email and send when ready, NOT just send emails. Is there a python wrapper for applescript that may work? (I don't know anything about applescript, so an example would help).

Answer: @ajrwhite adding attachments had one trick, you need to use 'Alias' from mactypes to convert a string/path object to a mactypes path. I'm not sure why but it works. here's a working example which creates messages with recipients and can add attachments: from appscript import app, k from mactypes import Alias from pathlib import Path def create_message_with_attachment(): subject = 'This is an important email!' body = 'Just kidding its not.' to_recip = ['myboss@mycompany.com', 'theguyih8@mycompany.com'] msg = Message(subject=subject, body=body, to_recip=to_recip) # attach file p = Path('path/to/myfile.pdf') msg.add_attachment(p) msg.show() class Outlook(object): def __init__(self): self.client = app('Microsoft Outlook') class Message(object): def __init__(self, parent=None, subject='', body='', to_recip=[], cc_recip=[], show_=True): if parent is None: parent = Outlook() client = parent.client self.msg = client.make( new=k.outgoing_message, with_properties={k.subject: subject, k.content: body}) self.add_recipients(emails=to_recip, type_='to') self.add_recipients(emails=cc_recip, type_='cc') if show_: self.show() def show(self): self.msg.open() self.msg.activate() def add_attachment(self, p): # p is a Path() obj, could also pass string p = Alias(str(p)) # convert string/path obj to POSIX/mactypes path attach = self.msg.make(new=k.attachment, with_properties={k.file: p}) def add_recipients(self, emails, type_='to'): if not isinstance(emails, list): emails = [emails] for email in emails: self.add_recipient(email=email, type_=type_) def add_recipient(self, email, type_='to'): msg = self.msg if type_ == 'to': recipient = k.to_recipient elif type_ == 'cc': recipient = k.cc_recipient msg.make(new=recipient, with_properties={k.email_address: {k.address: email}})