eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Following this question, I tried to run the following code to convert PDF with a path that contains Chinese characters to images: from pdf2image import convert_from_path images = convert_from_path('path with Chinese character in it/some Chinese character.pdf', 500) # save images  I got this error message: PDFPageCountError: Unable to get page count. I/O Error: Couldn't open file 'path with Chinese character in it/??????.pdf': No such file or directory.  in which all Chinese characters are replaced with "?". The issue is caused solely by the Chinese characters in the directory since the program worked as intended after I ensured that the path contains no Chinese characters. In pdf2image.py, I tried to alter the function pdfinfo_from_path, that out.decode("utf8", "ignore") is changed to e.g. out.decode("utf32", "ignore"), which also does not work. Not sure whether it is relevant: according to the aforementioned answer, I also need to install poppler. But my code also worked when the directory does not contain any Chinese characters. In addition, running this code conda install -c conda-forge poppler (from the answer above) never ends after several hours of waiting.

Answer: You could use the convert_from_bytes to avoid the issue: from pdf2image import convert_from_bytes with open('chinese_filename.pdf', 'rb') as f: images = convert_from_bytes(f.read(), 500)
Question: I have a simple question: why are my x-axis labels repeated?  Here's an MWE: X-Axis Labels MWE a = { # DATA -- 'CATEGORY': (VALUE, ERROR) 'Cats': (1, 0.105), 'Dogs': (2, 0.023), 'Pigs': (2.6, 0.134) } compositions = list(a.keys()) # MAKE INTO LIST a_vals = [i[0] for i in a.values()] # EXTRACT VALUES a_errors = [i[1] for i in a.values()] # EXTRACT ERRORS fig = plt.figure(figsize=(8, 6)) # DICTATE FIGURE SIZE bax = brokenaxes(ylims=((0,1.5), (1.7, 3)), hspace = 0.05) # BREAK AXES bax.plot(compositions, a_vals, marker = 'o') # PLOT DATA for i in range(0, len(a_errors)): # PLOT ALL ERROR BARS bax.errorbar(i, a_vals[i], yerr = a_errors[i], capsize = 5, fmt = 'red') # FORMAT ERROR BAR  Here's stuff I tried:  Manually setting x-axis tick marks using xticks Converting strings to floats using np.asarray(x, float) Reducing # ticks using pyplot.locator_params(nbins=3) 

Answer: You can use bax.locator_params(axis='x', nbins=len(compositions)) to reduce the number of x-ticks so that it matches the length of compositions. More on locator_params() method, which controls the behavior of major tick locators: https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.locator_params.html import matplotlib.pyplot as plt from brokenaxes import brokenaxes a = { # DATA -- 'CATEGORY': (VALUE, ERROR) 'Cats': (1, 0.105), 'Dogs': (2, 0.023), 'Pigs': (2.6, 0.134) } compositions = list(a.keys()) # MAKE INTO LIST a_vals = [i[0] for i in a.values()] # EXTRACT VALUES a_errors = [i[1] for i in a.values()] # EXTRACT ERRORS fig = plt.figure(figsize=(8, 6)) # DICTATE FIGURE SIZE bax = brokenaxes(ylims=((0, 1.5), (1.7, 3)), hspace=0.05) # BREAK AXES bax.plot(compositions, a_vals, marker='o') # PLOT DATA for i in range(0, len(a_errors)): # PLOT ALL ERROR BARS bax.errorbar(i, a_vals[i], yerr=a_errors[i], capsize=5, fmt='red') # FORMAT ERROR BAR bax.locator_params(axis='x', nbins=len(compositions)) plt.show()  Result:
Question: In PEP-616, the specification for removeprefix() includes this code block: def removeprefix(self: str, prefix: str, /) -> str: if self.startswith(prefix): return self[len(prefix):] else: return self[:]  Why does the last line say return self[:], instead of just return self?

Answer: [:] is an old idiom for copying sequences. Nowadays, we use the idiomatic .copy for lists; there isn't normally a good reason to copy strings, since they are supposed to be immutable, so the str class doesn't provide such a method. Furthermore, due to string interning, [:] may well return the same instance anyway. So, why include it in code like this? Because str can be subclassed. The clue is in the subsequent text:  When the arguments are instances of str subclasses, the methods should behave as though those arguments were first coerced to base str objects, and the return value should always be a base str.  Suppose we had a user-defined subclass: class MyString(str): ...  Notice what happens when we slice an instance to copy it: >>> type(MyString('xyz')[:]) <class 'str'>  In the example implementation, therefore, the [:] ensures that an instance of the base str type will be returned, conforming to the text specification.
Question: I have a CSV dataset like this. import pandas as pd from io import StringIO data=""" Date| Link April 1, 2009, 12:00 PM| 4 March 27, 2009, 12:00 PM| 8 April 29, 2009, 12:00 PM| 15 May 12, 2009, 12:00 PM| 9 June 9, 2009, 12:00 PM| 11 July 3, 2009, 12:00 PM| 329 June 16, 2009, 12:00 PM| 12 September 26, 2009, 12:00 PM| 48 October 4, 2009, 12:00 PM| 49 August 15, 2009, 12:00 PM| 10 November 30, 2009, 12:00 PM| 29 December 23, 2009, 12:00 PM| 68 April 1, 2009, 12:00 PM| 4 May 12, 2010, 12:00 PM| 9 September 26, 2012, 12:00 PM| 48 """ df = pd.read_csv(StringIO(data), delimiter='|')  Now I want to sort the dataset by the Date index; 1st day of calendar will need to appear first. For this, I have tried using df.sort_values(by = 'Date') but unfortunately, it gives me the sorted index using alphabetical order. How can I sort this data set by as like in calendar?

Answer: You can convert the Date column to datetime objects, and then use .sort_values: df["Date"] = pd.to_datetime(df["Date"]) df.sort_values("Date")  This outputs:  Date Link 1 2009-03-27 12:00:00 8 0 2009-04-01 12:00:00 4 12 2009-04-01 12:00:00 4 2 2009-04-29 12:00:00 15 3 2009-05-12 12:00:00 9 4 2009-06-09 12:00:00 11 6 2009-06-16 12:00:00 12 5 2009-07-03 12:00:00 329 9 2009-08-15 12:00:00 10 7 2009-09-26 12:00:00 48 8 2009-10-04 12:00:00 49 10 2009-11-30 12:00:00 29 11 2009-12-23 12:00:00 68 13 2010-05-12 12:00:00 9 14 2012-09-26 12:00:00 48
Question: I have a dataframe, example: df = [{'id': 1, 'text': 'text contains ok words'}, , {'id':2, 'text':'text contains word apple'}, {'id':3, 'text':'text contains words ok'}]  Example: keywords = ['apple', 'orange', 'lime']  And I want to check all columns 'text' to check if contains any word from my keywords, if so I want to alter that text column to: 'disconsider this case' I've tried to tokenize the column but then I'm not able to use the function I created to check, here is the example: df = pd.DataFrame(df) def remove_keywords(inpt): keywords = ['apple', 'orange', 'lime'] if any(x in word for x in keyword): return 'disconsider this case' else: return inpt df['text'] = df['text'].apply(remove_keywords) df  df['text'] = df.apply(lambda row: nltk.word_tokenize(row['text']), axis=1) for word in df['text']: if 'apple' in df['text']: return 'disconsider this case'  Any help appreciated. Thanks!!

Answer: this worked for me using pandas and a loop import pandas as pd keywords=['apple', 'orange', 'lime'] df = pd.DataFrame([{'id': 1, 'text': 'text contains ok words'}, {'id':2, 'text':'text contains word apple'}, {'id':3, 'text':'text contains words ok'}]) print(df) for i in range(len(df)): if any(word in df.iat[i,1] for word in keywords): df.iat[i,1]='discondider in this case' print(df)
Question: import sys import os import time MODULE_NAME = "mycode" def write_module(version): with open(MODULE_NAME+".py", "w") as f: f.write("def fun():"+os.linesep) f.write(" print('Code version:',"+str(version)+")") for i in range(5): # WRITE A PYTHON FILE AUTOMATICALLY write_module(i) # IMPORT IT if MODULE_NAME in sys.modules: del sys.modules[MODULE_NAME] # time.sleep(1) # <------------------------ WHY IS IT MANDATORY ???? module = __import__(MODULE_NAME) fun = module.fun # CALL IT fun()  Produces: Code version: 0 Code version: 0 Code version: 0 Code version: 0 Code version: 0  I expected: Code version: 0 Code version: 1 Code version: 2 Code version: 3 Code version: 4  I am developing Python code, writing Python code automatically. Python import instructions do not work as I expected. It looks like an asynchronous callback. I don't know why, adding the line time.sleep(1) corrects the error.

Answer: The source bytes are cached in __pycache__ directory. _validate_timestamp_pyc validates it against the source last-modified time — same without time.sleep(1) — and the source size. You can remove the pyc file before deleting the module from sys.modules.  if MODULE_NAME in sys.modules: os.remove(sys.modules[MODULE_NAME].__cached__) # Add this del sys.modules[MODULE_NAME]
Question: I have three python files (two that scrape data and 1 with my functions) I'm using in my app. Each file is about 150 lines, I don't want to include this code in views.py to keep my views as clean and readable as possible. Is there a best practice to keep things tidy? A separate folder inside the app with all external non-django .py files (similar to 'Templates' for .html or 'Static' for .css)? Any suggestions are appreciated. A Google search didn't yield any results.

Answer: You could create a separate app-- it depends on how sectioned off you want things. I prefer to just include the non-django .py files in the app that you expect to use them in the most. And then import the relevant functions in views.py from there. Sample of how the import would look in views.py for a function (called function) from a non-django .py file called operations.py located in the same app as the view: from .operations import function
Question: I am trying to create a Django app that creates a new database for every user when he/she signs up. I am going with this approach due to some reason. I have tried many ways using management commands and even Celery. But I am still getting the same error. 2022-12-23 07:16:07.410 UTC [49] STATEMENT: CREATE DATABASE tenant_asdadsad [2022-12-23 07:16:07,415: ERROR/ForkPoolWorker-4] Task user.utils.create_database[089b0bc0-0b5f-4199-8cf3-bc336acc7624] raised unexpected: ActiveSqlTransaction('CREATE DATABASE cannot run inside a transaction block\n') Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__ return self.run(*args, **kwargs) File "/app/user/utils.py", line 45, in create_database cursor.execute(f'CREATE DATABASE tenant_{tenant_id}') psycopg2.errors.ActiveSqlTransaction: CREATE DATABASE cannot run inside a transaction block  This is my task @shared_task def create_database(tenant_id): conn = psycopg2.connect(database="mydb", user="dbuser", password="mypass", host="db") cursor = conn.cursor() transaction.set_autocommit(True) cursor.execute(f'CREATE DATABASE tenant_{tenant_id}') cursor.execute(f'GRANT ALL PRIVILEGES ON DATABASE tenant_{tenant_id} TO dbuser') cursor.close() conn.close()  I have tried several ways but I always get the same error This is my API call def create(self, request, *args, **kwargs): serializer_class = mySerializer(data=request.data) if serializer_class.is_valid(): validated_data = serializer_class.validated_data or = validated_data["org"] or = Org.objects.create(**org) create_database.delay(str(or.id)) return Response(create_user(validated_data)) 

Answer: Use the autocommit property of the connection: from psycopg2 import sql def create_database(tenant_id): conn = psycopg2.connect(database="mydb", user="dbuser", password="mypass", host="db") cursor = conn.cursor() conn.autocommit = True #! # transaction.set_autocommit(True) #? dbname = sql.Identifier(f'tenant_{tenant_id}') create_cmd = sql.SQL('CREATE DATABASE {}').format(dbname) grant_cmd = sql.SQL('GRANT ALL PRIVILEGES ON DATABASE {} TO dbuser').format(dbname) cursor.execute(create_cmd) cursor.execute(grant_cmd) cursor.close() conn.close()  Read in the docs about connection.autocommit. Note also the use of the SQL string composition to avoid SQL injection.
Question: The following code appears to generate two mypy errors: Overloaded function signatures 1 and 3 overlap with incompatible return types and Overloaded function signatures 2 and 3 overlap with incompatible return types; but all overloads have different signatures - Literal[True], Literal[False] and None do not overlap. @overload def func_a(*, a: Literal[False] = ...) -> str: ... @overload def func_a(*, a: None = ...) -> str: ... @overload def func_a(*, a: Literal[True] = ...) -> int: ... def func_a(*, a: Optional[bool] = None) -> str | int: if a: return 1 return "foo" var1 = func_a() # str correctly discovered by VSCode Pylance var2 = func_a(a=False) # str correctly discovered by VSCode Pylance var3 = func_a(a=True) # int correctly discovered by VSCode Pylance  Why does Mypy think they overlap and how could I go about fixing this? Mypy version: 0.991 Python version: 3.11.1

Answer: The problem is that by writing = ... default values for every overload, you've marked the parameter as optional in every overload. A plain func_a() call matches every single overload of your function. You need to resolve that, so func_a() only matches one overload. Here's one way: @overload def func_a(*, a: Literal[False]) -> str: ... @overload def func_a(*, a: Literal[True]) -> int: ... @overload def func_a(*, a: None = None) -> str: ...  Here, only the None overload marks the parameter as optional, so func_a() only matches that overload. Alternatively, you could make the no-argument version its own overload: @overload def func_a(*, a: Literal[False]) -> str: ... @overload def func_a(*, a: Literal[True]) -> int: ... @overload def func_a(*, a: None) -> str: ... @overload def func_a() -> str: ...
Question: This code creates a list of 25 lists of 25 lists: vals = [] for i in range(25): vals.append([]) for j in range(25): vals[i].append([])  How could I translate this code to a single line instead of using 5 lines in Python?

Answer: You can use list_comprehension. res = [[[] for _ in range(25)] for _ in range(25)]  To check that result is the same, we can use numpy.ndarray.shape. >>> import numpy as np >>> np.asarray(vals).shape (25, 25, 0) >>> np.asarray(res).shape (25, 25, 0)
Question: I am writing Python in VSCode. I have a file with two functions, and a second file that calls those function. When I put breakpoints in one of the functions, the breakpoint does not hit. Here is test2.py that has the two functions... def func1(): for i in range(10): yield i def func2(): for i in range(10): print(i)  Here is test1.py for which the debugger is being launched from... import test2 test2.func1() test2.func2()  When I put a breakpoint on the for loop in both func1 and func2, then run the debugger from test1.py, the breakpoint for func1 is never hit, but the breakpoint in func2 is hit. Why is this?

Answer: You never actually advanced the generator, so it set up a generator instance for func1, but didn't begin executing it. If you want it to run out the generator, iterate the result of calling func1, e.g. replace: test2.func1()  with: # Extracts and prints all items for x in test2.func1(): print(x) # Or to extract and print a single item: print(next(test2.func1()))
Question: Quote from Python docs for Concatenate:  The last parameter to Concatenate must be a ParamSpec or ellipsis (...).  I know what ParamSpec is, but the ellipsis here drives me mad. It is not accepted by mypy: from typing import Callable, ParamSpec, Concatenate, TypeVar, Generic _P = ParamSpec('_P') _T = TypeVar('_T') class Test(Generic[_P, _T]): fn: Callable[Concatenate[_P, ...], _T]  E: Unexpected "..." [misc] E: The last parameter to Concatenate needs to be a ParamSpec [valid-type]  and is not explained anywhere in docs. PEP612 doesn't mention it. Is it just a mistake, appeared as a result of mixing Callable and Concatenate together? This issue is somewhat related and shows syntax with ellipsis literal in Concatenate:  The specification should be extended to allow either Concatenate[int, str, ...], or [int, str, ...], or some other syntax.  But this clearly targets "future syntax". Note: I'm aware of meaning of ellipsis as Callable argument, this question is specifically about Concatenate.

Answer: According to PEP-612's grammar, the ellipsis is not permitted in the Concatenate expression:  We now augment that with two new options: a parameter specification variable (Callable[P, int]) or a concatenation on a parameter specification variable (Callable[Concatenate[int, P], int]).  callable ::= Callable "[" parameters_expression, type_expression "]" parameters_expression ::= | "..." | "[" [ type_expression ("," type_expression)* ] "]" | parameter_specification_variable | concatenate "[" type_expression ("," type_expression)* "," parameter_specification_variable "]"   where parameter_specification_variable is a typing.ParamSpec variable, declared in the manner as defined above, and concatenate is typing.Concatenate.  However, the support for ellipsis as the last argument for Concatenate was introduced in April 2022 as part of Python 3.11. No type checker seems to handle this new case though.
Question: I have pandas DataFrame A. I am struggling transforming this into my desired format, see DataFrame B. I tried pivot or melt but I am not sure how I could make it conditional (string values to FIELD_STR_VALUE, numeric values to FIELD_NUM_VALUE). I was hoping you could point me the right direction. A: Input DataFrame |FIELD_A |FIELD_B |FIELD_C |FIELD_D | |--------|--------|--------|--------| |123123 |8 |a |23423 | |123124 |7 |c |6464 | |123144 |99 |x |234 |  B: Desired output DataFrame |ID |FIELD_A |FIELD_NAME |FIELD_STR_VALUE |FIELD_NUM_VALUE | |---|--------|-----------|----------------|----------------| |1 |123123 |B | |8 | |2 |123123 |C |a | | |3 |123123 |D | |23423 | |4 |123124 |B | |7 | |5 |123124 |C |c | | |6 |123124 |D | |6464 | |7 |123144 |B | |99 | |8 |123144 |C |x | | |9 |123144 |D | |234 | 

Answer: You can use: # dic = {np.int64: 'NUM', object: 'STR'} (df.set_index('FIELD_A') .pipe(lambda d: d.set_axis(pd.MultiIndex.from_arrays( [d.columns, d.dtypes], # or for custom NAMES #[d.columns, d.dtypes.map(dic)], names=['FIELD_NAME', None]), axis=1) ) .stack(0).add_prefix('FIELD_').add_suffix('_VALUE') .reset_index() )  NB. if you really want STR/NUM, map those strings from the dtypes (see comments in code). Output:  FIELD_A FIELD_NAME FIELD_int64_VALUE FIELD_object_VALUE 0 123123 FIELD_B 8.0 NaN 1 123123 FIELD_C NaN a 2 123123 FIELD_D 23423.0 NaN 3 123124 FIELD_B 7.0 NaN 4 123124 FIELD_C NaN c 5 123124 FIELD_D 6464.0 NaN 6 123144 FIELD_B 99.0 NaN 7 123144 FIELD_C NaN x 8 123144 FIELD_D 234.0 NaN
Question: If I have two dataframes: df1: df1 = pd.DataFrame({'A':[10,20,15,30,45], 'B':[17,33,23,10,12]}) A B 0 10 17 1 20 33 2 15 23 3 30 10 4 45 12    df2: df2 = pd.DataFrame({'C':['cat'], 'D':['dog'], 'E':['emu'], 'F':['frog'], 'G':['goat'], 'H':['horse'], 'I':['iguana']}) C D E F G H I 0 cat dog emu frog goat horse iguana    How do I combine the two dataframes and fill df1 whereby each row is a replicate of df2 ? Here is what I have so far. The code works as intended, but if I were to have hundreds of columns, then I would anticipate there would be a much easier way than my current method: Current Code: df1 = df1.assign(C = lambda x: df2.C[0], D = lambda x: df2.D[0], E = lambda x: df2.E[0], F = lambda x: df2.F[0], G = lambda x: df2.G[0], H = lambda x: df2.H[0], I = lambda x: df2.I[0])  Expected output:  A B C D E F G H I 0 10 17 cat dog emu frog goat horse iguana 1 20 33 cat dog emu frog goat horse iguana 2 15 23 cat dog emu frog goat horse iguana 3 30 10 cat dog emu frog goat horse iguana 4 45 12 cat dog emu frog goat horse iguana 

Answer: Use DataFrame.assign with Series by first row: df = df1.assign(**df2.iloc[0]) print (df) A B C D E F G H I 0 10 17 cat dog emu frog goat horse iguana 1 20 33 cat dog emu frog goat horse iguana 2 15 23 cat dog emu frog goat horse iguana 3 30 10 cat dog emu frog goat horse iguana 4 45 12 cat dog emu frog goat horse iguana
Question: I am generating this series of numbers using a for loop [1.e-03 1.e-04 1.e-05 1.e-06 1.e-07 1.e-08 1.e-09 1.e-10 1.e-11 1.e-12]  This is the for loop: alphas = np.zeros(10) alphas[0] = 0.001 for i in range(1,10): alphas[i] = alphas[i-1] * 0.1  My heart of hearts tells me this is not "pythonic", but my brain can't come up a list comprehension to build this. I've tried numpy.linspace, arange, etc, but can't quite land where I need to. I wrote the for loop in 60 seconds, but am trying every time I write a for loop to think about how I could do it with a list comprehension.

Answer: Think about what you want the range on: it's powers of 10 [10**x for x in range(-3, -13, -1)]  Or 10.0**np.arange(-3, -13, -1) # or 10.0**-np.arange(3, 13)  The numpy example uses a float 10.0 because in numpy,  Integers to negative integer powers are not allowed  Try it online
Question: I have an enum.StrEnum, for which I want to add attributes to the elements. For example: class Fruit(enum.StrEnum): APPLE = ("Apple", { "color": "red" }) BANANA = ("Banana", { "color": "yellow" }) >>> str(Fruit.APPLE) "Apple" >>> Fruit.APPLE.color "red"  How can I accomplish this? (I'm running Python 3.11.0.) This question is not a duplicate of this one, which asks about the original enum.Enum.

Answer: The answer is much the same as in the other question (but too long to leave in a comment there): from enum import StrEnum class Fruit(StrEnum): # def __new__(cls, value, color): member = str.__new__(cls, value) member._value_ = value member.color = color return member # APPLE = "Apple", "red" BANANA = "Banana", "yellow"  If you really want to use a dict for the attributes you can, but you run the risk of forgetting an attribute and then one or more of the members will be missing it.
Question: My VS code suggestions will not display any documentation next to the suggestions. What settings do I need to change? I have editor>suggest>show inline details active but it is doing it. Here is mine:  I want it to look like this:  I am using python. I tried searching through the settings. I tried looking it up on the internet. I tried looking at these GitHub issues: https://github.com/microsoft/vscode/issues/18582 https://github.com/microsoft/vscode/pull/25812 https://github.com/microsoft/vscode/issues/26282

Answer: It seems that the space is too small. This setting has been deprecated. You can use the following alternatives: Use command "ctrl + space" when you get intellisense:  And this is a switch. You only need to use this command once, and you can get the prompt of this window in the future.