eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have been working with dictionaries that I have to modify within different parts of my code. I am trying to make sure if I do not miss anything about there is no need for dict_update() in any scenario. So the reasons to use update() method is either to add a new key-value pair to current dictionary, or update the value of your existing ones. But wait!? Aren't they already possible by just doing: >>>test_dict = {'1':11,'2':1445} >>>test_dict['1'] = 645 >>>test_dict {'1': 645, '2': 1445} >>>test_dict[5]=123 >>>test_dict {'1': 645, '2': 1445, 5: 123}  In what case it would be crucial to use it ? I am curious. Many thanks

Answer: 1. You can update many keys on the same statement. my_dict.update(other_dict)  In this case you don't have to know how many keys are in the other_dict. You'll just be sure that all of them will be updated on my_dict. 2. You can use any iterable of key/value pairs with dict.update As per the documentation you can use another dictionary, kwargs, list of tuples, or even generators that yield tuples of len 2. 3. You can use the update method as an argument for functions that expect a function argument. Example: def update_value(key, value, update_function): update_function([(key, value)]) update_value("k", 3, update_on_the_db) # suppose you have a update_on_the_db function update_value("k", 3, my_dict.update) # this will update on the dict
Question: when trying to import - from PIL import Image this error occurs: ERROR: Could not find a version that satisfies the requirement PIL (from versions: none) ERROR: No matching distribution found for PIL 

Answer: the solution: pip install Pillow  after that make sure it works: from PIL import Image
Question: why does print("Lorem" and "aliqua" in string ) Gives True. A Boolean, But print("Lorem" or "aliqua" in string ) Gives 'Lorem'. A String string = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua" print("Lorem" and "aliqua" in string ) >>> True print("Lorem" or "aliqua" in string ) >>> Lorem 

Answer: Try: print("Lorem" in string and "aliqua" in string )  And print("Lorem" in string or "aliqua" in string )  Explanation: The condition in string will always be true as it checks string is non empty. >>> if "harsha": ... print("hi") ... hi >>> if "": ... print("hi") ... <<No output>>
Question: I am trying to pass date and id through url, but getting an error, I have passed just id before and I usually do it like this. path('user_payment_menu/<int:pk>/',user_payment_menu, name='user_payment_menu'),  but now I want date to pass after int:pk/ but when I add date after a slash I am getting an error.

Answer: Probably the easiest way to define a date is with a custom path converter. You can implement this with: # app_name/converters.py class DateConverter: regex = '\d{4}-\d{1,2}-\d{1,2}' format = '%Y-%m-%d' def to_python(self, value): return datetime.strptime(value, self.format).date() def to_url(self, value): return value.strftime(self.format) Then you can register the format and use the <date:…> path converter: # app_name/urls.py from django.urls import path, register_converter from app_name.converters import DateConverter from app_name.views import user_payment_menu register_converter(DateConverter, 'date') urlpatterns = [ path('user_payment_menu/<int:pk>/<date:mydate>/',user_payment_menu, name='user_payment_menu'), then in the view you define an extra attribute that will contain the date as a date object: # app_name/views.py def user_payment_menu(request, pk, mydate): # … You can use a date object when generating a URL, for example with: {% url 'user_payment_menu' pk=somepk mydate=somedate %}
Question: I used a function in Python/Numpy to solve a problem in combinatorial game theory. import numpy as np from time import time def problem(c): start = time() N = np.array([0, 0]) U = np.arange(c) for _ in U: bits = np.bitwise_xor(N[:-1], N[-2::-1]) N = np.append(N, np.setdiff1d(U, bits).min()) return len(*np.where(N==0)), time()-start problem(10000)  Then I wrote it in Julia because I thought it'd be faster due to Julia using just-in-time compilation. function problem(c) N = [0] U = Vector(0:c) for _ in U elems = N[1:length(N)-1] bits = elems .⊻ reverse(elems) push!(N, minimum(setdiff(U, bits))) end return sum(N .== 0) end @time problem(10000)  But the second version was much slower. For c = 10000, the Python version takes 2.5 sec. on an Core i5 processor and the Julia version takes 4.5 sec. Since Numpy operations are implemented in C, I'm wondering if Python is indeed faster or if I'm writing a function with wasted time complexity. The implementation in Julia allocates a lot of memory. How to reduce the number of allocations to improve its performance?

Answer: The original code can be re-written in the following way: function problem2(c) N = zeros(Int, c+2) notseen = falses(c+1) for lN in 1:c+1 notseen .= true @inbounds for i in 1:lN-1 b = N[i] ⊻ N[lN-i] b <= c && (notseen[b+1] = false) end idx = findfirst(notseen) isnothing(idx) || (N[lN+1] = idx-1) end return count(==(0), N) end  First check if the functions produce the same results: julia> problem(10000), problem2(10000) (1475, 1475)  (I have also checked that the generated N vector is identical) Now let us benchmark both functions: julia> using BenchmarkTools julia> @btime problem(10000) 4.938 s (163884 allocations: 3.25 GiB) 1475 julia> @btime problem2(10000) 76.275 ms (4 allocations: 79.59 KiB) 1475  So it turns out to be over 60x faster. What I do to improve the performance is avoiding allocations. In Julia it is easy and efficient. If any part of the code is not clear please comment. Note that I concentrated on showing how to improve the performance of Julia code (and not trying to just replicate the Python code, since - as it was commented under the original post - doing language performance comparisons is very tricky). I think it is better to concentrate in this discussion on how to make Julia code fast.  EDIT Indeed changing to Vector{Bool} and removing the condition on b and c relation (which mathematically holds for these values of c) gives a better speed: julia> function problem3(c) N = zeros(Int, c+2) notseen = Vector{Bool}(undef, c+1) for lN in 1:c+1 notseen .= true @inbounds for i in 1:lN-1 b = N[i] ⊻ N[lN-i] notseen[b+1] = false end idx = findfirst(notseen) isnothing(idx) || (N[lN+1] = idx-1) end return count(==(0), N) end problem3 (generic function with 1 method) julia> @btime problem3(10000) 20.714 ms (3 allocations: 88.17 KiB) 1475
Question: How expose metrics in multiprocess app use start_http_server I found many examples with gunicorn in internet but i want use start_http_server what should i do with code below to make it work properly? from multiprocessing import Process import time, os from prometheus_client import start_http_server, multiprocess, CollectorRegistry, Counter MY_COUNTER = Counter('my_counter', 'Description of my counter') os.environ["PROMETHEUS_MULTIPROC_DIR"] = "tmp" def f(): print("+1") MY_COUNTER.inc() if __name__ == '__main__': start_http_server(8000) p = Process(target=f, args=()) a = p.start() p2 = Process(target=f, args=()) p2.start() time.sleep(1) print("collect") registry = CollectorRegistry() data = multiprocess.MultiProcessCollector(registry) while True: time.sleep(1) 

Answer: I was figuring out the same thing, and the solution was as simple as you would imagine. Updated your example code to work:  from multiprocessing import Process import shutil import time, os from prometheus_client import start_http_server, multiprocess, CollectorRegistry, Counter COUNTER1 = Counter('counter1', 'Incremented by the first child process') COUNTER2 = Counter('counter2', 'Incremented by the second child process') COUNTER3 = Counter('counter3', 'Incremented by both child processes') def f1(): while True: time.sleep(1) print("Child process 1") COUNTER1.inc() COUNTER3.inc() def f2(): while True: time.sleep(1) print("Child process 2") COUNTER2.inc() COUNTER3.inc() if __name__ == '__main__': # ensure variable exists, and ensure defined folder is clean on start prome_stats = os.environ["PROMETHEUS_MULTIPROC_DIR"] if os.path.exists(prome_stats): shutil.rmtree(prome_stats) os.mkdir(prome_stats) # pass the registry to server registry = CollectorRegistry() multiprocess.MultiProcessCollector(registry) start_http_server(8000, registry=registry) p = Process(target=f1, args=()) a = p.start() p2 = Process(target=f2, args=()) p2.start() print("collect") while True: time.sleep(1)  localhost:8000/metrics # HELP counter1_total Incremented by the first child process # TYPE counter1_total counter counter1_total 9.0 # HELP counter2_total Incremented by the second child process # TYPE counter2_total counter counter2_total 9.0 # HELP counter3_total Incremented by both child processes # TYPE counter3_total counter counter3_total 18.0
Question: I’m working on a python project with a notebook and .env file on VsCode. I have problem when trying to refresh environment variables in a notebook (I found a way but it's super tricky). My project: .env file with: MY_VAR="HELLO_ALICE" test.ipynb file with one cell: from os import environ print('MY_VAR = ', environ.get('MY_VAR'))  What I want:  set the env variable and run my notebook (see HELLO_ALICE) edit .env file: change "HELLO_ALICE" to "HELLO_BOB" set the env variable and run my notebook (see HELLO_BOB)  What do not work:  open my project in vsCode, open terminal in terminal run: >> set -a; source .env; set +a; open notebook, run cell --> I see HELLO_ALICE edit .env (change HELLO_ALICE TO HELLO_BOB) restart notebook (either click on restart or close tab and reopen it) in terminal run: >> set -a; source .env; set +a; (same as step 2) open notebook, run cell --> I see HELLO_ALICE  So I see twice HELLO_ALICE instead of HELLO_ALICE then HELLO_BOB... But if it was on .py file instead of notebook, it would have worked (I would see HELLO_ALICE first then HELLO_BOB) To make it work: Replace step 5. by: Close VsCode and reopen it Why it is a problem: It is super tricky. I'm sure that in 3 month I will have forgotten this problem with the quick fix and I will end up loosing again half a day to figure out what is the problem & solution. So my question is: Does anyone know why it works like this and how to avoid closing and reopening VsCode to refresh env variable stored in a .env file on a notebook ? (Closing and reopening VsCode should not change behavior of code) Notes:  VsCode version = 1.63.2 I tired to use dotenv module and load env variable in my notebook (does not work) question: How to set env variable in Jupyter notebook works only if you define your env variables inside notebook this behavior happen only on env variables. For instance if instead a .env file I use a env.py file where i define my env constants as python variables, restarting the notebook will refresh the constants. 

Answer: The terminal you open in VSC is not the same terminal ipython kernel is running. The kernel is already running in an environment that is not affected by you changing variables in another terminal. You need to set the variables in the correct environment. You can do that with dotenv, but remember to use override=True. This seems to work: $ pip3 install python-dotenv  import dotenv from os import environ env_file = '../.env' f = open(env_file,'w') f.write('MY_VAR="HELLO_ALICE"') f.close() dotenv.load_dotenv(env_file, override=True) print('MY_VAR = ', environ.get('MY_VAR')) f = open(env_file,'w') f.write('MY_VAR="HELLO_BOB"') f.close() dotenv.load_dotenv(env_file, override=True) print('MY_VAR = ', environ.get('MY_VAR'))  MY_VAR = HELLO_ALICE MY_VAR = HELLO_BOB
Question: I'm using FastAPI to create backend for my project. I have a method that allows to upload a file. I implemented it as follows: from fastapi import APIRouter, UploadFile, File from app.models.schemas.files import FileInResponse router = APIRouter() @router.post("", name="files:create-file", response_model=FileInResponse) async def create(file: UploadFile = File(...)) -> FileInResponse: pass  As you can see, I use a dedicated pydantic model for a method result—FileInResponse: from pathlib import Path from pydantic import BaseModel class FileInResponse(BaseModel): path: Path  And I follow this naming pattern for models (naming models as <Entity>InCreate, <Entity>InResponse, and so on) throughout the API. However, I couldn't create a pydantic model with a field of the type File, so I had to declare it directly in the route definition (i.e. without a model containing it). As a result, I have this long auto generated name Body_files_create_file_api_files_post in the OpenAPI docs:  Is there a way to change the schema name?

Answer: In case someone is interested into renaming auto generated fields from FastAPI in openapi.json file, you should add an operation_id param to your route. This will allow FastAPI to use this id to generate a clearer attribute name for the generated schema. Before: @router.post( "/{opportunity_id}/files", status_code=status.HTTP_201_CREATED, ) async def attach_opportunity_file( db: Database, uploaded_file: UploadFile = File(title="File to upload"), ) -> OpportunityFile: pass   After: @router.post( "/{opportunity_id}/files", status_code=status.HTTP_201_CREATED, operation_id="attach_opportunity_file", # New operation_id added ) async def attach_opportunity_file( db: Database, uploaded_file: UploadFile = File(title="File to upload"), ) -> OpportunityFile: pass
Question: I saw on this tweet that Google Colab move to Python 3.7 on February 2021. As of today however (January 2022), Python 3.10 is out, but Colab still runs Python 3.7. My (voluntarily) naive take is that this is quite a significant lag. Why are they not at least on Python 3.8 or even 3.9? Is it simply to make sure that some compatibility criteria are met?

Answer: The only reason is that they want to have the most compatible version of Python worldwide. According to the Python Readiness report (Python 3.7 Readiness), version 3.7 supports almost 80.6% of the most used packages so far. Still, this coverage is 78.3% for version 3.8, 70.6% for version 3.9, and 49.7% for version 3.10 (as of March 29, 2022). They still use this version today if Python 3.6 was not in its EOL. Lucky us, python.org decided to get rid of versions below 3.7. 😊 On the other hand, you can update the Python version in your Colab by running some Linux commands in the notebook. However, whenever you start a new notebook, Google ignores the updates and returns to the original version. Google should provide options for selecting the Python version. Because of this, I do not use Colab in most cases, especially when teaching Python to my students. Update (January 12, 2023): Now, Google Colaboratory supports Python 3.8.16 Python 3.8 Readiness. After a long time, we see some improvement. But it's still outdated because the current version is 3.11.1. The Python Readiness report says 80.8% of most used packages support Python 3.8, and 30.6% support 3.11. But we know this information comes from info on PyPi. In practice, this support is far more than what the package maintainer says in the repository. Many packages support 3.11, but they still mention the lower version of Python. The reason is that the maintainer hasn't had a chance to check and update their production yet. Update (May 07, 2024): Fortunately, we have seen more activities on the Google server admins recently 😉. It has been around three months since Google's admins updated Python to version 3.10 (Python 3.10 Readiness), and the current version that colab supports by default is 3.10.12, which is still old (the current stable version of Python is 3.12.3)). The funny part is that they didn't update it at least to 3.10.14, the current version of Python 3.10. "Half a loaf is better than none."
Question: I am using MongoDB to store the results of a script into a database. When I want to reload the data back into python, I need to decode the JSON (or BSON) string into a pydantic basemodel. With a pydantic model with JSON compatible types, I can just do: base_model = BaseModelClass.parse_raw(string)  But the default json.loads decoder doesn't know how to deal with a DataFrame. I can overwrite the .parse_raw function into something like: from pydantic import BaseModel import pandas as pd class BaseModelClass(BaseModel): df: pd.DataFrame class Config: arbitrary_types_allowed = True json_encoders = { pd.DataFrame: lambda df: df.to_json() } @classmethod def parse_raw(cls, data): data = json.loads(data) data['df'] = pd.read_json(data['df']) return cls(**data)  But ideally I would want to automatically decode fields of type pd.DataFrame rather than manually change the parse_raw function every time. Is there any way of doing something like:  class Config: arbitrary_types_allowed = True json_encoders = { pd.DataFrame: lambda df: df.to_json() } json_decoders = { pd.DataFrame: lambda df: pd.read_json(data['df']) }  To make the detection of any field which should be a data frame, be converted to one, without having to modify the parse_raw() script?

Answer: Pydantic V2: You can define a custom data type and specify a serializer which will automatically handle conversions: from typing import Annotated, Any from pydantic import BaseModel, GetCoreSchemaHandler import pandas as pd from pydantic_core import CoreSchema, core_schema class myDataFrame(pd.DataFrame): @classmethod def __get_pydantic_core_schema__( cls, source_type: Any, handler: GetCoreSchemaHandler ) -> CoreSchema: validate = core_schema.no_info_plain_validator_function(cls.try_parse_to_df) return core_schema.json_or_python_schema( json_schema=validate, python_schema=validate, serialization=core_schema.plain_serializer_function_ser_schema( lambda df: df.to_json() ), ) @classmethod def try_parse_to_df(cls, value: Any): if isinstance(value, str): return pd.read_json(value) return value # Create a model with your custom type class BaseModelClass(BaseModel): df: myDataFrame # Create your model sample_df = pd.DataFrame([[1, 2], [3, 4], [5, 6], [7, 8]], columns=["A", "B"]) my_model = BaseModelClass(df=sample_df) # Should also be able to parse from json my_model = BaseModelClass(df=sample_df.to_json()) # Even more dramatically my_model_2 = BaseModelClass.model_validate_json(my_model.model_dump_json())
Question: I made a geopandas dataframe and I want to use geopandas_dataframe.explore() to create an interactive map. Here is my code. First I create the geopandas dataframe, I check the dtypes and I try to map the dataframe with gdf.explore(). Unfortunately, my code just finishes without errors and no map is shown. code: geometry = [Point(xy) for xy in zip(df[1], df[0])] gdf = geopandas.GeoDataFrame(df, geometry=geometry) print(gdf.head()) print(gdf.dtypes) gdf.explore()  output:  0 1 geometry 0 51.858306 5.778404 POINT (5.77840 51.85831) 1 51.858322 5.778410 POINT (5.77841 51.85832) 2 51.858338 5.778416 POINT (5.77842 51.85834) 3 51.858354 5.778422 POINT (5.77842 51.85835) 4 51.858370 5.778429 POINT (5.77843 51.85837) 0 float64 1 float64 geometry geometry dtype: object Process finished with exit code 0  Why don't I get a map? I already tried gdf.show() but that doesn't exist. What do I need to do to show the geopandas map?

Answer: What IDE are you using? In Jupyter Notebook your code (slightly modified) works for me. However, when I run it in PyCharm I get, "Process finished with exit code 0" with no plot. import geopandas as gpd import pandas as pd from shapely.geometry import Point data_dict = {'x': {0: -110.1, 1: -110.2, 2: -110.3, 3: -110.4, 4: -110.5}, 'y': {0: 40.1, 1: 40.2, 2: 40.3, 3: 40.4, 4: 40.5}} df = pd.DataFrame(data_dict) geometry = [Point(xy) for xy in zip(df['x'], df['y'])] gdf = gpd.GeoDataFrame(df, geometry=geometry, crs=4326) print(gdf.head()) print(gdf.dtypes) gdf.explore()   Edit: Looks like you can save your folium figure to a html. This worked for me from PyCharm. m = gdf.explore() outfp = r"<your dir path>\base_map.html" m.save(outfp)
Question: For example, ss is an sklearn.preprocessing.StandardScaler object. If ss is fitted already, I want to use it to transform my data. If ss is not fitted yet, I want to use my data to fit it and transform my data. Is there a way to know whether ss is already fitted or not?

Answer: Sklearn implements the check_is_fitted function to check if any generic estimator is fitted, which works with StandardScaler: from sklearn.preprocessing import StandardScaler from sklearn.utils.validation import check_is_fitted ss = StandardScaler() check_is_fitted(ss) # Raises error ss.fit([[1,2,3]]) check_is_fitted(ss) # No error
Question: I've created this map using geopandas, but I can't make the color bar have the same size as the figure. ax = covid_death_per_millon_geo.plot(column = 'total_deaths_per_million', legend = True, cmap = 'RdYlGn_r', figsize=(20,15)) ax.set_title('Covid deaths per Million', size = 20) ax.set_axis_off()  https://i.sstatic.net/a26oJ.png

Answer: The colorbars on GeoPandas plots are Matplotlib colorbar objects, so it's worth checking those docs. Add a legend_kwds option and define the shrink value to change the colorbar's size (this example will shrink it to 50% of the default): ax = covid_death_per_millon_geo.plot( column="total_deaths_per_million", legend=True, legend_kwds={ "shrink":.5 }, cmap="RdYlGn_r", figsize=(20, 15) )  Alternatively, change the colorbar's location to put it below the map. You may want to change both the location and the size: ax = covid_death_per_millon_geo.plot( column="total_deaths_per_million", legend=True, legend_kwds={ "location":"bottom", "shrink":.5 }, cmap="RdYlGn_r", figsize=(20, 15) )
Question: I'm trying to run a Jupyter notebook on Ubuntu 21.10. I've installed python, jupyter notebook, and all the various prerequisites. I added export PATH=$PATH:~/.local/bin to my bashrc so that the command jupyter notebook would be operational from the terminal. When I call jupyter notebook from the terminal, I get the following error message from my browser: Access to the file was denied. The file at /home/username/.local/share/jupyter/runtime/nbserver-260094-open.html is not readable. It may have been removed, moved, or file permissions may be preventing access.  I'm using the latest version of FireFox. I've read a number of guides on this and it seems to be a permissions error, but none of the guides that I've used have resolved the issue. Using sudo does not help, in fact it causes Exception: Jupyter command "jupyter-notebook" not found. to be thrown. That being said, I am still able to access the notebook server. If I go to the terminal and instead click on the localhost:8888 or IP address of the notebook server then it takes me to the notebook and everything runs without issue. I would like to solve this so that when I run jupyter notebook I'm taken to the server and don't need to go back to the terminal window and click the IP address. It's inconvenient and can slow me down if I'm running multiple notebooks at once. Any help on this issue would be greatly appreciated!

Answer: I had the same problem. Ubuntu 20.04.3 LTS Chromium Version 96.0.4664.110 This was the solution in my case: Create the configuration file with this command: jupyter notebook --generate-config  Edit the configuration file ~/.jupyter/jupyter_notebook_config.py and set: c.NotebookApp.use_redirect_file = False  Make sure that this configuration parameter starts at the beginning of the line. If you leave one space at the beginning of the line, you will get the message that access to the file was denied. Otherwise you can clean and reinstall JupyterLab jupyter lab clean --all pip3 install jupyterlab --force-reinstall