eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In Python, it's known that the most efficient way to create a list with n repetitions of the same element (let's say the string 's') is by using list multiplication, as shown below: lst = ['s'] * 1000  However, when the list is non-empty initially, what would be the most optimal method to append the same element n times? Here are a couple of methods that come to mind: Method1: lst = [1,2,3] for _ in range(1000): lst.append('s')  Method2: lst = [1,2,3] lst.extend(['s'] * 1000) # or # lst.extend(['s' for _ in range(1000)])  But it's worth noting that Method 2 does create a temporary long list, e.g. ['s' for _ in range(1000)]. Are there any alternative approaches that are more efficient, both in terms of time complexity and space usage? Or among the existing methods, which one is deemed the most efficient?

Answer: I propose: def addition(N): lst = [1, 2, 3] return lst+['s']*N  I profiled the approaches: import itertools from performance_measurement import run_performance_comparison def method1(N): lst = [1, 2, 3] for _ in range(N): lst.append("s") return lst def method2(N): lst = [1, 2, 3] lst.extend(["s"] * N) return lst def generator_approach(N): #@user2390182 lst = [1, 2, 3] lst.extend("s" for _ in range(N)) return lst def addition(N): lst = [1, 2, 3] return lst + ["s"] * N def itertools_approach(N): #@juanpa.arrivillaga lst = [1, 2, 3] lst.extend(itertools.repeat("s", N)) return lst approaches = [method1, method2, generator_approach, addition, itertools_approach] for approach in approaches[1:]: data = [100] assert approach(*data) == approaches[0](*data) run_performance_comparison( approaches, [ 1_000, 3_000, 5_000, 10_000, 30_000, 100_000, 300_000, 500_000, 1_000_000, ], title="Performance Comparison", number_of_repetitions=10, )   It seems to me ['s']*N is very performant, but itertools beats it. Profiling code: import timeit from functools import partial import matplotlib.pyplot as plt from typing import List, Dict, Callable from contextlib import contextmanager @contextmanager def data_provider(data_size, setup=lambda N: N, teardown=lambda: None): data = setup(data_size) yield data teardown() def run_performance_comparison(approaches: List[Callable], data_size: List[int], setup=lambda N: [N], teardown=lambda: None, number_of_repetitions=5, title='Performance Comparison', data_name='N'): approach_times: Dict[Callable, List[float]] = {approach: [] for approach in approaches} for N in data_size: with data_provider(N, setup, teardown) as data: for approach in approaches: function = partial(approach, *data) approach_time = min(timeit.Timer(function).repeat(repeat=number_of_repetitions, number=1)) approach_times[approach].append(approach_time) for approach in approaches: plt.plot(data_size, approach_times[approach], label=approach.__name__) plt.yscale('log') plt.xscale('log') plt.xlabel(data_name) plt.ylabel('Execution Time (seconds)') plt.title(title) plt.legend() plt.show()
Question: I have the following code to create a tuple contains multiple tuples with integer pairs: iterable = ( tuple(zip([0, 1, 2], _)) for _ in product(range(9), repeat=3) ) next(iterable) # First element is not needed print(list(iterable)) # This code produces: [((0, 0), (1, 0), (2, 1)), ... , ((0, 8), (1, 8), (2, 8))]  But I need that if last element of a tuple is "0" (e.g. (0, 0) or (2, 0)), I have to remove that tuple. So new list should be like this: [((2, 1),), ... , ((1, 2), (2, 7)), ((1, 2), (2, 8)), ... , ((0, 8), (1, 8), (2, 8))]  I actually achieved this goal by the following code but it is not the correct way I think, I don't know: x = () for i in iterable: y = () for j in i: if j[-1] != 0: y += (j,) x += (y,) print(list(x))  How can I do this with itertools module and in one line, if possible? If needed, I can change the code at the top of this question, to create the desired list in one line. Thank you.

Answer: Use filter() to remove the elements ending in 0 from the result of zip(). iterable = ( tuple(filter(lambda x: x[-1] != 0, zip([0, 1, 2], _))) for _ in product(range(9), repeat=3) )
Question: Are UML class representations different for each programming language, or do they adhere to a standard? I have conducted extensive research but remain confused. For instance:  In Python: set_name(): None In Java: setName(): void In Kotlin: setName(): Unit 

Answer: There is not one "standard" for UML naming conventions. I would advise you use names similar to what you would use in the programming language you are using. As long as your intended reader understands the material then it is a good convention. Most important is that whatever you choose remains consistent throughout your documentation. See: this question as well.
Question: I have a file with a data like this: ------------------------------ ------------------------------ <TIME:2020-01-01 01:25:10> <TIME:2020-01-01 01:25:10> <TIME:2020-01-01 01:25:10> <TIME:2020-01-01 01:25:10> ------ ++++++ %%RequestHandler DATA1 = 123456 ERROR1 = 500 DATA2 = 56789 ERROR2 = 505 Count = 4 ---  I would like to create a dataframe like     DATA1 ERROR1     123456 500   56789 505    

Answer: Another regex approach with pivot: import re # or file.read() out = (pd.DataFrame(re.findall(r'^\s+(\w+)(\d+) = (\d+)', text, flags=re.M)) .pivot(index=1, columns=0, values=2) .rename_axis(index=None, columns=None) ) print(out)  Output:  DATA ERROR 1 123456 500 2 56789 505  Used input: text = '''------------------------------ ------------------------------ <TIME:2020-01-01 01:25:10> <TIME:2020-01-01 01:25:10> <TIME:2020-01-01 01:25:10> <TIME:2020-01-01 01:25:10> ------ ++++++ %%RequestHandler DATA1 = 123456 ERROR1 = 500 DATA2 = 56789 ERROR2 = 505 Count = 4'''  regex demo
Question: I am new to python and am coming from C# and java. I want to instantiate a class of the type provided as type hint R as following from typing import (TypeVar, Generic) class BaseParams(object): def __init__(self) -> None: self.name = 'set-in-base-class' class ChildParams(BaseParams): def __init__(self) -> None: super().__init__() self.name = 'set-in-child-class' R = TypeVar('R', bound= BaseParams) class MyGeneric(Generic[R]): def __init__(self) -> None: super().__init__() def test(self): r = R() # how should I instantiate R here print(r.name) c = MyGeneric[ChildParams]() c.test()  something like the following C# code class BaseParams { public BaseParams() { Name = "set-in-base-class"; } public string Name { get; set; } } class ChildParams : BaseParams { public ChildParams() { Name = "set-in-child-class"; } } class MyGenericClass<R> where R : BaseParams, new() { public void test() { var r = new R(); Console.WriteLine(r.Name); } }  I've made quite a lot search on how to do that in python and all the sources refer to a situation where we provide the type in a method or something like that. I wonder if it is possible to do that at all. would you please someone help me to have a workaround on this?

Answer: It's still type hints and not type declaration, but I like inspecting weird objects and looking at their insides, so I gave it a go: I tried to inspect Generics in interactive session using dir and also looked for clues in the typing's source. My testing consisted of having one MyGeneric without R specified and one with, then inspecting them both. Both of them had __orig_bases__ which included Generic[R], but then I noticed that the parametrized MyClass has additional __orig_class__ attribute - bingo! >>> a = MyGeneric() >>> b = MyGeneric[int]() >>> a.__orig_bases__ (typing.Generic[~R],) >>> b.__orig_ b.__orig_bases__ b.__orig_class__( >>> b.__orig_bases__ (typing.Generic[~R],) >>> a.__orig_class__ Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'MyGeneric' object has no attribute '__orig_class__'. Did you mean: '__orig_bases__'? >>> b.__orig_class__ __main__.MyGeneric[int]  By looking at Generic's source earlier, I already knew that .__args__ parameter gets whatever is in [], as tuple (because it could be multiple things). So: >>> b.__orig_class__.__args__ (<class 'int'>,) >>> b.__orig_class__.__args__[0] <class 'int'>  Going back to your code I had in file, I put that attribute chain and run the file - and got set-in-child-class. from typing import (TypeVar, Generic) class BaseParams(object): def __init__(self) -> None: self.name = 'set-in-base-class' class ChildParams(BaseParams): def __init__(self) -> None: super().__init__() self.name = 'set-in-child-class' R = TypeVar('R', bound= BaseParams) class MyGeneric(Generic[R]): def __init__(self) -> None: super().__init__() def test(self): r = self.__orig_class__.__args__[0]() print(r.name) c = MyGeneric[ChildParams]() c.test()  I'd advise to put that magical chain of attributes as some function with nice name. Like get_parametrized_base_arg or something. Or just put a good comment to explain the magic.
Question: ChatGPT is the newest platform for running Python in a Jupyter-like environment. However the installed libraries are limited. You cannot access the internet too. So, I cannot use pip to install. How can I install a new library?

Answer: You can upload a wheel file taken from PyPI, select cp38-manylinux_..._x86_64.whl and upload it. Tell ChatGPT to unzip and move it to /home/sandbox/.local/lib/python3.8/site-packages/ Then you can import and use it normally. Here's an example case where I install DuckDB: https://chat.openai.com/share/fa3df390-8a25-45d3-997a-c19d4f19df67
Question: I created two Gmail accounts and I'm trying to create an e-mail thread between them with the Python Gmail API. I can send e-mails without any issue, but when it comes to replying to each other and creating a thread, it is simply not working : the new message is successfully displaying as the answer of the received email for the sender, but it is appearing as a new message - without a linked thread - for the receiver. This problem was described here in 2019 : https://stackoverflow.com/a/63186609/21966625 However, the Gmail APIs changed a lot since this article and I didn't find how to use these advices with today's API. I tried to carefully respect the instructions of the docs by defining the message's parameters References and In-Reply-To as the received message's id when replying. Indeed, I retrieve the email : received_email= service.users().messages().get(userId='me', id=label['id']).execute()  I get a dict looking like: {'id': '189462395f418017', 'threadId': '189462395f418017', 'labelIds': ['UNREAD','INBOX'], 'snippet': 'xxx'....}  Hence, when I'm building my e-mail, the following method should work : message_id=received_email['id'] message = EmailMessage() message.set_content('') message['To'] = 'john.doe@gmail.com' message['From'] = 'john.doe@gmail.com' message['References'] = message_id message['In-Reply-To'] = message_id message['Subject'] = 'Automated draft'  In the same way, I defined the threadId as the id of the message I wanted to reply to. create_message = {'raw': encoded_message, 'threadId': message_id } send_message = (service.users().messages().send(userId="me", body=create_message).execute())  Thanks to this part of the code, the answers are correctly displayed (for the sender of the answer) as explained above, but it appears as a new message - unlinked to a thread - for the receiver.

Answer: Actually I found why my method did not work ; even if the dict mention a kind of message id : email = {'id': '189462395f418017', 'threadId': '189462395f418017', 'labelIds': ['UNREAD','INBOX'], 'snippet': 'xxx'....}  I thought the messageIDcould be taken just by call email['id']. The real messageID is somewhere in the ['payload']['headers'] dictionnary ; one could find it by a loop like : for p in email['payload']['headers']: if p["name"] == "Message-Id": message_id = p['value']  This way we have the true messageID of the email, and the threads are successfully created.
Question: I am getting an ImportError while using GPTSimpleVectorIndex from the llama-index library. Have installed the latest version of llama-index library and trying to run it on python 3.9. from llama_index import GPTSimpleVectorIndex, SimpleDirectoryReader, LLMPredictor, PromptHelper, ServiceContext ImportError: cannot import name 'GPTSimpleVectorIndex' from 'llama_index' (E:\Experiments\OpenAI\data anaysis\llama-index-main\venv\lib\site-packages\llama_index\__init__.py  The source code is given below, import os, streamlit as st from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader, LLMPredictor, PromptHelper, ServiceContext from langchain.llms.openai import OpenAI 

Answer: Try use GPTVectorStoreIndex instead of GPTSimpleVectorIndex: from llama_index import GPTVectorStoreIndex, ..
Question: I am trying to extract the link inside a href but all I am finding it is the text inside the element The website code is the following: <div class="item-info-container "> <a href="/imovel/32600863/" role="heading" aria-level="2" class="item-link xh-highlight" title="Apartamento T3 na avenida da Liberdade, São José de São Lázaro e São João do Souto, Braga"> Apartamento T3 na avenida da Liberdade, São José de São Lázaro e São João do Souto, Braga </a>  And the code I am using is: element_handle = page.locator('//div[@class="item-info-container "]//a').all_inner_texts()  No matter if I specify //a[@href] or not, my output is always the title text: Apartamento T3 na avenida da Liberdade, São José de São Lázaro e São João do Souto, Braga  When what I really want to achieve is: /imovel/32600863/  Any ideas of where my logic is failing me?

Answer: Using get_attribute: link = page.locator('.item-info-container ').get_by_role('link').get_attribute('href')  More than one locator: link_locators = page.locator('.item-info-container ').get_by_role('link').all() for _ in link_locators: print(_.get_attribute('href'))
Question: I have a function that sometimes call with Numba as a device function to execute on the GPU and sometimes I call directly from within regular Python on the host: def process(): # perform computation process_cuda = cuda.jit(device=True)(process)  sometimes I call process() from directly from Python, and sometimes I invoke the process_cuda wrapper as a kernel with Numba. My question, how can I tell, from within the process function, if it was called directly from Python or if it is executing as a Numba device function?

Answer: Numba offers a function called cuda.current_context(), if the code is executed from a GPU the function return the current CUDA context otherwise it will return None when run using a CPU So, we can initialize a variable in the process() to check if the code is executed from a CPU or GPU def process(): cuda_context = cuda.current_context() if cuda_context: return 'GPU' else: return 'CPU'  If there is anything I did wrong, please share it with me.
Question: I encountered an issue I am confused about with Python modules. I have built a minimal working example: $ tree ├── app │ ├── configuration.py │ ├── error_code.py └── test.py  app/configuration.py from error_code import ErrorCode class Configuration: def status(self): return self._status def __init__(self): self._status = ErrorCode.Error1  app/error_code.py import enum class ErrorCode(enum.Enum): Success = 0 Error1 = 1  test.py #!/usr/bin/env python from app.configuration import Configuration from app.error_code import ErrorCode c = Configuration() print(c.status()) print(c.status() == ErrorCode.Error1)  Command: PYTHONPATH=app ./test.py Output: ErrorCode.Error1 False  Why does in such case the output show ErrorCode.Error1 while it says False for the equal check? I suppose it has to do with the module, because if in app/configuration.py I replace from error_code import ErrorCode with from app.error_code import ErrorCode it works as expected. But it seems weird to me. Where should I look to get the documentation for this specific behavior?

Answer: It's because the directory that test.py is in is also added to the import path. Which is why this import in test.py worked to begin with: from app.error_code import ErrorCode  However, when you do: from error_code import ErrorCode  in app/configuration.py it relied on PYTHONPATH=app/. But the original import was cached as "app.error_code" in sys.modules, so when you import error_code it isn't found in the cache, because it would need to exist as "error_code", and it isn't recognized as the same module, so it is reimported as a new module. IMO, you should use: from app.error_code import ErrorCode  And as you noted, the import will be appropriately cached. Really, it's the parent directory of app that should be added to the PYTHONPATH to begin with. Better yet, you shouldn't be relying on the working directory or on PYTHONPATH, and just make your project pip installable.
Question: My default assumption was that the chunk_size parameter would set a ceiling on the size of the chunks/splits that come out of the split_text method, but that's clearly not right: from langchain.text_splitter import RecursiveCharacterTextSplitter, CharacterTextSplitter chunk_size = 6 chunk_overlap = 2 c_splitter = CharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap) text = 'abcdefghijklmnopqrstuvwxyz' c_splitter.split_text(text)  prints: ['abcdefghijklmnopqrstuvwxyz'], i.e. one single chunk that is much larger than chunk_size=6. So I understand that it didn't split the text into chunks because it never encountered the separator. But so then the question is what is the chunk_size even doing? I checked the documentation page for langchain.text_splitter.CharacterTextSplitter here but did not see an answer to this question. And I asked the "mendable" chat-with-langchain-docs search functionality, but got the answer "The chunk_size parameter of the CharacterTextSplitter determines the maximum number of characters in each chunk of text."...which is not true, as the code sample above shows.

Answer: CharacterTextSplitter will only split on separator (which is '\n\n' by default). chunk_size is the maximum chunk size that will be split if splitting is possible. If a string starts with n characters, has a separator, and has m more characters before the next separator then the first chunk size will be n if chunk_size < n + m + len(separator). Your example string has no matching separators so there's nothing to split on. Basically, it attempts to make chunks that are <= chunk_size, but will still produce chunks > chunk_size if the minimum size chunks that can be created are > chunk_size.
Question: I have a list containing ~50k elements of a custom data type (the latter is probably not important for my question) I'm sorting the list using pythons builtin list.sort() method. myList: List[Foo] = ... myList.sort(key=Foo.x)  Since the sorting takes a couple of minutes, I would like to have a progress bar for the sorting process. I haven't found any solutions online. Is this even possible? I'm aware sorting algorithms may be complex and it might not be possible to measure the sorting progress at all. However, it would be fine for my usecase to have a "rough" measurement, like 25%, 50%, 75%...

Answer: Given the interface provided by sort, you don't have too many options for hooking into the actual sorting algoithm. However, if 50K keys is slow, it is most likely that invoking the key function is slow, which is computed prior to the actual sorting. From the docs:  The key corresponding to each item in the list is calculated once and then used for the entire sorting process.  So if you keep count of how many times the key method is invoked, you could gain a rough estimate for the whole sorting process. To do so, you could create a wrapper for the key function to manage this book keeping: def progress_sort(data, *, key=lambda v: v, on_increment=None): total = len(data) if on_increment is None: start = time.time() def on_increment(c): print(f"{time.time() - start}: {c/total * 100}%") count = 0 def progress_key(val): nonlocal count if count % int(total / 10) == 0: on_increment(count) count += 1 return key(val) data.sort(key=progress_key) on_increment(total)  Example with some dummy data and a slow key method def slow_key(val): time.sleep(1.0/500_000) return val data = [random.randint(-50_000, 50_000)/1.0 for i in range(50_000)] progress_sort(data, key=slow_key)  0.0: 0.0% 0.5136210918426514: 10.0% 1.0435900688171387: 20.0% 1.6074442863464355: 30.0% 2.156496524810791: 40.0% 2.9734878540039062: 50.0% 3.4794368743896484: 60.0% 4.016523599624634: 70.0% 4.558118104934692: 80.0% 5.047779083251953: 90.0% 5.545809030532837: 100.0%  This method could then be combined with whatever type of library you wish to use for updating status. You may wish to further configure the data supplied to the provided hooks, however, the principle remains the same. Here's an example that uses tqdm: def slow_key(val): time.sleep(1.0/500_000) return val data = [random.randint(-50_000, 50_000)/1.0 for i in range(50_001)] with tqdm(total=len(data), desc="sorting") as pbar: progress_sort(data, key=slow_key, on_increment=lambda c: pbar.update(c - pbar.n)) pbar.set_description("Finished")  sorting: 80%|███████▉ | 40000/50001 [00:05<00:01, 5802.30it/s]  Finished: 100%|██████████| 50001/50001 [00:07<00:00, 6489.14it/s]
Question: I have a dataframe like this     Column A Column B     Hello [{id: 1000, abbreviatedId: 1, name: “John", planet: “Earth”, solarsystem: “Milky Way”, universe: “this one”, continent: {id: 33, country: “China", Capital: “Bejing”}, otherId: 400, language: “Cantonese”, species: 23409, creature: “Human”}]   Bye [{id: 2000, abbreviatedId: 2, name: “James", planet: “Earth”, solarsystem: “Milky Way”, universe: “this one”, continent: {id: 33, country: “Russia", Capital: “Moscow”}, otherId: 500, language: “Russian”, species: 12308, creature: “Human”}]     How do I iterate through the rows of the dataframe to drop all rows with country: "China" before writing to external location? I have tried if df.select(array_contains(col("columnb.continent.country"), "China")) != True: df.write.format("delta").mode("overwrite").save("file://path/")  and for row in df.rdd.collect(): if df.select(array_contains(col("columnb.continent.country"), "China")) != True: df.drop(row) df.write.format("delta").mode("overwrite").save("file://path/") 

Answer: One way is using exists array function. from pyspark.sql.functions import expr from pyspark.sql import Row df = spark.createDataFrame([ [ [ Row(**{"id": 1000, "abbreviatedId": 1, "name": "John", "planet": "Earth", "solarsystem": "Milky Way", "universe": "this one", "continent": Row(**{"id": 33, "country": "China", "Capital": "Bejing"}), "otherId": 400, "language": "Cantonese", "species": 23409, "creature": "Human"}), Row(**{"id": 1001, "abbreviatedId": 2, "name": "Alex", "planet": "Mars", "solarsystem": "Milky Way", "universe": "this one", "continent": Row(**{"id": 34, "country": "Japan", "Capital": "Tokyo"}), "otherId": 400, "language": "Japanese", "species": 23409, "creature": "Human"}) ] ]], ["b"]) df.filter(expr("not exists(b, x -> x.continent.country == 'China')"))  The syntax Row(**dict) will create an instance of Row through argument unpacking.
Question: Where does the py_modules parameter (I don't know what it is called) go in a pyproject.toml file? [project] name = "elements" version = "1.0.0" description = "Magic" readme = "README.md" requires-python = ">=3.10" license = {file = "LICENSE"} authors = [ {name = "x"}, {name = "y"} ] maintainers = [ {name = "x"}, {name = "y"} ] [project.scripts] stuff = "stuff:__init__" [tool.setuptools] packages = ["baseclasses", "elements"] # this raises error. py_modules = ["__init__"] [build-system] requires = ["setuptools>=43.0.0", "wheel"] build-backend = "setuptools.build_meta"  My project structure looks like this: . ├── .gitignore ├── LICENSE ├── README.md ├── __init__.py ├── baseclasses │ ├── __init__.py │ ├── bunch_of_py_files.py ├── elements │ ├── __init__.py │ ├── bunch_of_py_files.py ├── pyproject.toml  I have tried to put it under the project section and the tool.setuptools section, but both raised the project must not contain {'py_modules'} properties error. I was looking through setuptools and pip docs, but the only thing I could find was for setup.py files, which I read was being deprecated and as such didn't want to use.

Answer: I was able to find the solution here [tool.setuptools] packages = ["baseclasses", "elements"] py-modules = ["__init__"] # dash, not underscore