eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm running Ubuntu Server and upgraded it from 22.10 to 23.04. After the installation finished, I tried running one of my python scripts and one of my modules went missing. I tried to reinstall it with pip3 install playsound but I get this error: error: externally-managed-environment × This environment is externally managed ╰─> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install. If you wish to install a non-Debian-packaged Python package, create a virtual environment using python3 -m venv path/to/venv. Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make sure you have python3-full installed. If you wish to install a non-Debian packaged Python application, it may be easiest to use pipx install xyz, which will manage a virtual environment for you. Make sure you have pipx installed. See /usr/share/doc/python3.11/README.venv for more information. note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.  As the error says, I tried to install playsound with sudo apt install python3-playsound but it's not available through apt. I also tried python3 -m pip install playsound and that gives the same error. I don't really know what else to do because I have never needed to mess with pip like this before. Any thoughts/ideas appreciated!

Answer: Figured it out, just needed to activate the virtual environment I made.
Question: I have a GitHub Action that calls a python script: - name: Test Python Script run: | python python_script.py  This runs fine and prints me the logs from within here: if __name__ == "__ main __": print('Calling function') func_call() print('End of script')  However, I don't get the print statements from func_call itself: def func_call() : print('inside func_call')  When I run from my own terminal, the output is: Calling function inside func_call End of script  Yet when I run from a GitHub Action I only get: Calling function End of script  Is there a way I can propagate all print statements to the GitHub Action logs?

Answer: The solution was relatively simple, just adding the -u flag: - name: Test Python Script run: | python -u python_script.py  From man python:  -u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.
Question: When fitting a linear GAM model in python imposing n_splines=5, a piecewise-linear function is fitted: import statsmodels.api as sm from pygam import LinearGAM data = sm.datasets.get_rdataset('mtcars').data Y = data['mpg'] X = data.drop("mpg",axis=1) model = LinearGAM(spline_order=1,n_splines=5).fit(X, Y)  By using .coef from fitted model, the coefficientes for every splines can be recovered for further analysis: model.coef_  However, how can we obtain the sections of each of the 5 splines for each variable? As an example, for cyl variable we would fit the following splines:  The 5 sections are determined by the knots, so, in the plot we would see the variable limits for the computed betas. (i.e.:4-5,5-6,6-7,7-8). The only thing I find in the documentation the method model.edge_knots which is  array-like of floats of length 2. The minimum and maximum domain of the spline function.  In this example it corresponds for cyl to [4,8].

Answer: Finally I have come up with a solution, in this one I use partial dependence to calculate the function with its slope changes. In this one I take double differences and with it the change of slope.     XX = model_gam.generate_X_grid(term=i) pdep, confi = model_gam.partial_dependence(term=i, X=XX, width=0.95) first_diff = [float("{:.3f}".format(i)) for i in np.diff(pdep)] second_diff = abs(np.diff(first_derivative)) values_list = XX[np.where(second_diff > 0)[0],i]    This leads to this result which is suboptimal:    But seems a good enough first apporach.
Question: How to type (in Python, e.g. for MyPy) a function that expects two parameters - an enum and one of its values/members? from enum import Enum from typing import TypeVar, Type class MyEnumA(Enum): A = 1 B = 2 class MyEnumB(Enum): A = 1 B = 2 TE = TypeVar('TE', bound=Enum) def myfunction(member: TE, e: Type[TE]) -> None: pass myfunction(MyEnumA.A, MyEnumA) # all right myfunction(MyEnumA.A, MyEnumB) # I expect mypy-error here but it passed print(type(MyEnumA.A)) # says: <enum 'MyEnumA'> print(type(MyEnumB.A)) # says: <enum 'MyEnumB'> print(f'{isinstance(MyEnumA.A, MyEnumA)=}') # says: isinstance(MyEnumA.A, MyEnumA)=True print(f'{isinstance(MyEnumA.A, MyEnumB)=}') # says: isinstance(MyEnumA.A, MyEnumB)=False reveal_type(MyEnumA) # mypy: Revealed type is "def (value: builtins.object) -> e.MyEnumA" reveal_type(MyEnumA.A) # mypy: Revealed type is "Literal[e.MyEnumA.A]?"  I would like to understand  Why MyPy does not state an error for the second call; and How to type myfunction so that MyPy detects an error there.  More examples: myfunction(MyEnumA.A, MyEnumA) # should pass - member of enum myfunction(MyEnumB.A, MyEnumB) # should pass - member of enum myfunction(MyEnumA.A, MyEnumB) # should fail - member of other enum myfunction(MyEnumB.A, MyEnumA) # should fail - member of other enum 

Answer: The key observation is what MyPy actually considers a class MyEnumA to be: reveal_type(MyEnumA) # mypy: Revealed type is "def (value: builtins.object) -> e.MyEnumA"  It gives a clue to annotate myfunction in the following way, no matter how unobvious it is: P = ParamSpec('P') def myfunction(member: TE, e: Callable[P, TE]) -> None:  That makes MyPy to find typing error in the second call of myfunction: error: Argument 2 to "myfunction" has incompatible type "Type[MyEnumA]"; expected "Callable[[object], MyEnumB]" [arg-type]
Question: I have a long file that is composed of many blocks like this: x=0 abc 20 def 76 ghi 11 x=1 def 45 ghi 32 x=2 jkl 56 mno 76 abc 134 . . . x=393 xyz 21 abc 9 x=394 ghi 43 def 166 xyz 25  I need the regular expression (for python) to get this output: return_list = [20,'x',134,...,9,'x']  (so matches for abc and x should be assigned to the same group) So I need to look in each block if abc is available, if it is match integer after abc, if not match the 'x' at the beginning. Block length (lines) are different for each block. I tried that regexp: (?ms)(?=x=\d).*?(?=\n\n) That matches all blocks separately. I now need the extension to search for abc in the individual blocks and return the x if not present. I have tried many things but since I am a beginner I can not find a solution. I hope someone can help me.

Answer: You can achieve the result you want with a single regex, by changing your regex to terminate at either abc = \d+ or \n\n or \Z (to allow a match of the last group at the end of the string - this is required for your sample data as it doesn't finish with \n\n). (?ms)(?=x=\d).*?(?:abc (\d+)|\n\n|\Z)  If abc = \d+ is present in the block, the digits will be captured in group 1, otherwise it will be empty. You can then use this fact to substitute the value with an 'x': res = [m.group(1) or 'x' for m in re.finditer(r'(?ms)(?=x=\d).*?(?:abc (\d+)|\n\n|\Z)', data)]  Output: ['20', 'x', '134', '9', 'x']  Note if you want integers rather than strings you can modify the code slightly: res = [int(m.group(1)) if m.group(1) else 'x' for m in re.finditer(r'(?ms)(?=x=\d).*?(?:abc (\d+)|\n\n|\Z)', data)]  Output: [20, 'x', 134, 9, 'x']
Question: Setting a variable with a match can be done simply like this: Mode = 'fast' puts = '' match Mode: case "slow": puts = 'v2 k5' case "balanced": puts = 'v3 k5' case "fast": puts = 'v3 k7'  But can you do something like this? Mode = 'fast' puts = match Mode: case "slow": 'v2 k5' case "balanced": 'v3 k5' case "fast": 'v3 k7'  Currently it results in a syntax error.

Answer: Not as far as I know in python. In this case I wouldn't use pattern matching at all: puts = { "slow": "v2 k5", "balanced": "v3 k5", "fast": "v3 k7" }.get(Mode, default)  Clearer, more declarative, and no repeating yourself. But of course you can't use clever pattern destructuring.
Question: In polars, what is the way to make a copy of a dataframe? In pandas it would be: df_copy = df.copy()  But what is the syntax for polars?

Answer: That would be clone : df = pl.DataFrame( { "a": [1, 2, 3, 4], "b": [0.5, 4, 10, 13], "c": [True, True, False, True], } ) df_copy = df.clone() #cheap deepcopy/clone  Output : ​ print(df_copy) ​ shape: (4, 3) ┌─────┬──────┬───────┐ │ a ┆ b ┆ c │ │ --- ┆ --- ┆ --- │ │ i64 ┆ f64 ┆ bool │ ╞═════╪══════╪═══════╡ │ 1 ┆ 0.5 ┆ true │ │ 2 ┆ 4.0 ┆ true │ │ 3 ┆ 10.0 ┆ false │ │ 4 ┆ 13.0 ┆ true │ └─────┴──────┴───────┘
Question: I want to create a self hosted LLM model that will be able to have a context of my own custom data (Slack conversations for that matter). I've heard Vicuna is a great alternative to ChatGPT and so I made the below code: from llama_index import SimpleDirectoryReader, LangchainEmbedding, GPTListIndex, \ GPTSimpleVectorIndex, PromptHelper, LLMPredictor, Document, ServiceContext from langchain.embeddings.huggingface import HuggingFaceEmbeddings import torch from langchain.llms.base import LLM from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM !export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512 class CustomLLM(LLM): model_name = "eachadea/vicuna-13b-1.1" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name) pipeline = pipeline("text2text-generation", model=model, tokenizer=tokenizer, device=0, model_kwargs={"torch_dtype":torch.bfloat16}) def _call(self, prompt, stop=None): return self.pipeline(prompt, max_length=9999)[0]["generated_text"] def _identifying_params(self): return {"name_of_model": self.model_name} def _llm_type(self): return "custom" llm_predictor = LLMPredictor(llm=CustomLLM())  But sadly I'm hitting the below error: OutOfMemoryError: CUDA out of memory. Tried to allocate 270.00 MiB (GPU 0; 22.03 GiB total capacity; 21.65 GiB already allocated; 94.88 MiB free; 21.65 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF  Here's the output of !nvidia-smi (before running anything): Thu Apr 20 18:04:00 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1 | |-----------------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA A10G Off| 00000000:00:1E.0 Off | 0 | | 0% 23C P0 52W / 300W| 0MiB / 23028MiB | 18% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | No running processes found | +---------------------------------------------------------------------------------------+  Any idea how to modify my code to make it work?

Answer: length is too long, 9999 will consume huge amount of GPU RAM, especially using 13b model. try 7b model. And try using something like peft/bitsandbytes to reduce GPU RAM usage. set load_in_8bit=True is a good start.
Question: I was looking into how to create an index in Django. I came across two ways:  Class Student(models.Model): name= models.CharField(........, db_index=True) age= models.CharField(........, db_index=True) Class Student(models.Model): name= models.CharField(........) age= models.CharField(........) Class Meta: indexes = [models.Index(name="first_index",fields=["name","age"],)]  Can someone clarify what the difference between them is? Moreover, in the db_index=true method, is there a single index for all the fields with db_index=true ?

Answer: The latter will make a combined index, so an index that takes the two fields, and makes a retrieval if both fields are specified efficient. This means that if only name is given to filter on, the index will (likely) not be useful, if you only specify the age, this will be the same, only if you specify both name and age that index in case two will be useful. It will however be more useful if you specify both when you have an index on the combination and it thus can easily find the Student(s) with that name and age. Otherwise it will probably look in both indexes, and then make a set of records that appear in both. The two are however not mutually exclusive, you can make an index for name, for age, and for name and age combined with: class Student(models.Model): name= models.CharField(max_length=128, db_index=True) age = models.IntegerField(db_index=True) class Meta: indexes = [models.Index(name='name_age_index', fields=['name','age'],)]
Question: I'm working on a Python package where I need to get credentials (email and password) from the user via CLI and they can enter their credentials just as follows. $ my_package auth --email abc@efg.com --password testpass123  My package is responsible for storing and using the provided credentials in the next calls (even if the system reboots). What is the best way of implementing this? Using environment variables? Online password managers? Keeping them in the user's $HOME directory?

Answer: In memory: These days most operating systems have memory protection which prevents other processes to access the memory used by some other process(you may specifically ask for sharing the memory though). So even a Python variable could do the job(just get it from sys.argv and store it in a variable) or an environment variable but you need to make sure your code doesn't make use of insecure calls like loading a pickle, shelves, or using exec, eval etc from unknown source. While the process is running, Python variable and environment variable basically have same security. If one can read that variable, can read env variable by os.environ. The difference is that if your script forks another process, environment variables are gonna get copied to the child. In file system: Since you mentioned "after a reboot" you need to store the credentials in the file system. There is a well-known package called keyring. It helps you store your credential encrypted. If I'm not wrong it uses the logged-in user's password to encrypt the data. It has a friendly interface to use. You can use the client's email as the username for setup. There is nothing wrong with online password managers but if you have one locally and the system has set permissions properly it saves you one extra request.
Question: My problem: I want to check if the provided word is a common English word. I'm using pyenchant currently to see if a word is an actual word but I can't find a function in it that returns the frequency of a word/if it's a common word. Example code: import enchant eng_dict = enchant.Dict("en_US") words = ['hello', 'world', 'thisisntaword', 'anachronism'] good_words = [] for word in words: if eng_dict.check(word): # currently this checks if it's an english word, but I also want it to check it it's commonly used word good_words.append(word) print(good_words)  What it returns as is: ['hello', 'world', 'anachronism']. What I want it to return:['hello', 'world'] because anachronism is obviously not a common word. Any solutions my problem?

Answer: You could use the Google Ngram API for this: url = "https://books.google.com/ngrams/json" query_params = { "content": <my_noun_phrase/string of noun phrases>, "year_start": 2017, "year_end": 2019, "corpus": 26, "smoothing": 1, "case_insensitive": True } response = requests.get(url=url, params=query_params)  This API lets you access v3 of the Google ngram database, which is the most recent version available. Note, however, that the API is not officially documented, and since you run into rate limits quite easily, it's not production-proof. Alternative tools are PhraseFinder (https://phrasefinder.io/) and NGRAMS (https://ngrams.dev/). PhraseFinder is a wrapper around v2 of the Google ngram database; NGRAM is a wrapper around v3 of the same database. They are both free and can handle more traffic than the Google API.
Question: I have a subclass of dict: class MyDict(dict): pass  Later I use the definition: my_var: MyDict[str, int] = {'a': 1, 'b': 2}  MyPy complains: error: "MyDict" expects no type arguments, but 2 given [type-arg]  How can I define MyDict so that MyPy recognizes it as generic with two type arguments? I have tried deriving from typing.Dict and adding protocol MutableMapping, both to no avail.

Answer: As a general rule you can remember to always provide type arguments to generic types, regardless of context. (Remember: explicit is better than implicit.) This means in annotations (e.g. d: dict[str, int]) and when subclassing (e.g. class MyDict(dict[str, int]): ...). If you want to retain genericity in terms of some or all type parameters of the base class in your own subclass, then pass type variables in place of specific type arguments: from collections.abc import Hashable from typing import TypeVar K = TypeVar("K", bound=Hashable) V = TypeVar("V") class MyDict(dict[K, V]): ... class MyStrKeyDict(dict[str, V]): ... class MyIntValDict(dict[K, int]): ... x: MyDict[str, int] y: MyStrKeyDict[int] z: MyIntValDict[str]  The reason MyPy told you it did not expect any type arguments for MyDict is that it implicitly assumed MyDict is a subclass of dict[Any, Any] because you did not specify the type arguments. Side note: It would still be wrong to now assign e.g. x = {"a": 1} as you did in your example because dict is not a subtype of MyDict. This is why it is usually preferable to subclass the ABC (in this case Mapping or MutableMapping) or use protocols.
Question: I am exploring FastAPI, and got it working on my Docker Desktop on Windows. Here's my main.py which is deployed successfully in Docker: #main.py import fastapi import json from fastapi.responses import JSONResponse app = fastapi.FastAPI() @app.get('/api/get_weights1') async def get_weights1(): weights = {'aa': 10, 'bb': 20} return json.dumps(weights) @app.get('/api/get_weights2') async def get_weights2(): weights = {'aa': 10, 'bb': 20} return JSONResponse(content=weights, status_code=200)  And I have a simple python file get_weights.py to make requests to those 2 APIs: #get_weights.py import requests import json resp = requests.get('http://127.0.0.1:8000/api/get_weights1') print('ok', resp.status_code) if resp.status_code == 200: print(resp.json()) resp = requests.get('http://127.0.0.1:8000/api/get_weights2') print('ok', resp.status_code) if resp.status_code == 200: print(resp.json())  I get the same responses from the 2 APIs, output: ok 200 {"aa": 10, "bb": 20} ok 200 {'aa': 10, 'bb': 20}  The response seems the same whether I use json.dumps() or JSONResponse(). I've read the FastAPI documentation on JSONResponse, but I still have below questions: May I know if there is any difference between the 2 methods? If there is a difference, which method is recommended (and why?)?

Answer: In FastAPI you can create response 3 different ways (from most concise to most flexible): 1 return dict # Or model or ...  In docs you linked we can see that FastAPI will automatically stringify this dict and wrap it in JSONResponse. This way is most concise and covers majority of use cases. 2 However sometimes you have to return custom headers (for example REMOTE-USER=username) or different status code (maybe 201 - Created or 202 - Accepted). This case you need to use JSONResponse. return JSONResponse(content=dict) # Here we need to have dict.  Problem is that now if we don't have simple dict, we have to use jsonable_encoder(some_model) # -> dict to get it. So it's a tad more verbose. For available options check Starlette documentation, since FastAPI just reexports it. More complex example: return JSONResponse(content=jsonable_encoder(some_model), status_code=201, headers={"REMOTE-USER": username})  3 Finally you do not need to return json - you can also return csv, html or any other type of file. This case we have to use Response and specify media_type. Likewise use Starlette docs. return Response('Hello, world!', media_type='text/plain')  In short Note that Fastapi documentation states:  When you return a Response directly its data is not validated, converted (serialized), nor documented automatically.  So we see what is difference: first method has good integration with other FastAPI functionality, so should always be preferred. Use second option only if you need to provide custom headers or status codes. Finally, use third option only if you want to return something that is not json.
Question: I have a small problem with my date code. I would like the date to appear in Spanish add the date in Spanish. The date is currently formatted like this: 2004/03/30 But I want the formatting to be like this: 04/21/2023 What I'm trying to do is display any date in Spanish, as follows: Viernes, 02 De Junio De 2023 Is there any way to do this? The code I am using is the following: from datetime import datetime, timedelta s = '2004/03/30' date = datetime.strptime(s, "%B") modified_date = date + timedelta(days=1) print(datetime.strftime(modified_date, "%B")) 

Answer: You can use the babel library to localize the date to Spanish. pip install babel  The code would look like this: from datetime import datetime from babel.dates import format_date s = '2004/03/30' date = datetime.strptime(s, "%Y/%m/%d") # Spanish locale spanish_date = format_date(date, "EEEE, dd 'de' MMMM 'de' yyyy", locale='es') print(spanish_date)  Output: martes, 30 de marzo de 2004
Question: I want to draw a 3D picture with these data using matplotlib and Axes3D import numpy as np import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D data = np.array([[4244.95, 4151.69, 2157.41, 829.769, 426.253, 215.655], [8263.14, 4282.98, 2024.68, 1014.6, 504.515, 250.906], [8658.01, 4339.53, 2173.56, 1087.65, 544.069, 544.073]]) x = np.array([1, 2, 4, 8, 16, 32]) y = np.array([2, 4, 8]) x, y = np.meshgrid(x, y) z = data fig = plt.figure() ax = Axes3D(fig) ax.plot_surface(x, y, z, rstride=1, cstride=1, cmap='rainbow') ax.set_xlabel('Stride') ax.set_ylabel('Bitwidth') ax.set_zlabel('Performance') plt.show()  In my computer, it gives a blank picture. But I run this code in other two computers, one is correct, the other one is blank. matplotlib: 3.7.1 numpy: 1.24.2 I have tried in windows11 and wsl ubuntu-20.04, but still the blank pictures.

Answer: replace this line ax = Axes3D(fig)  with ax = fig.add_subplot(projection='3d')  you do this in order to alert Matplotlib that we're using 3d data.