eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: So, I have a Collection of documents (e.g. Person) structured in this way: class Person(Document): name = StringField(max_length=200, required=True) nationality = StringField(max_length=200, required=True) earning = ListField(IntField())  when I save the document I only input the name and nationality fields because this is the information. Then, every now and then, I want to update the the earning of each person of a particular nationality. Let's imagine that there is some formula that allows me to compute the earning field (e.g. I query some magical api called EarningAPI that returns the earning of a person given its name). To update them I would do something like: japanese_people = Person.objects(Q(nationality='Japanese'))).all() for japanese_person in japanese_people: japanese_person.earning.append(EarningAPI(japanese_person.name)) Person.objects.insert(japanese_people, load_bulk=False)  The EarningAPI has also the possibility to work in batches, so that i can give a list of names and it returns a list of earning(s) (one for each name). This method is far faster and less expensive. Is the one by one way correct? What is the best way to take advantage of the batches? Thanks

Answer: Using method from Mongoengine bulk update without objects.update(): from pymongo import UpdateOne from mongoengine import Document, ValidationError class Person(Document): name = StringField(max_length=200, required=True) nationality = StringField(max_length=200, required=True) earning = ListField(IntField()) japanese_people = Person.objects(Q(nationality='Japanese')).all() japanese_ids = [person.id for person in japanese_people] earnings = EarningAPI(japanese_ids) # I'm assuming it takes a list of id's as input and returns a list of earnings. bulk_operatons = [ UpdateOne( {'_id': j_id}, {'$set': {'earning': earn}}, upsert=True ), for j_id, earn in zip(japanese_ids, earnings) ] result = Person._get_collection().bulk_write(bulk_operations, ordered=False)  I can't be certain if this is faster than the one by one method because I don't have access to your magic API to benchmark, but this should be the way to do it by batch.
Question: There's the requests and urllib page that can access http(s) protocol pages in Python, e.g. import requests requests.get('stackoverflow.com')  but when it comes to chrome:// pages, e.g. chrome://settings/help, the url libraries won't work and this: import requests requests.get('chrome://settings/help')  throws the error: InvalidSchema: No connection adapters were found for 'chrome://settings/help'  I guess there's no way for requests or urllib to determine which chrome to use and where's the binary executable file for the browser. So the adapter can't be easily coded. The goal here is to pythonically obtain the string Version 111.0.5563.146 (Official Build) (x86_64) from the chrome://settings/help page of the default chrome browser on the machine, e.g. it looks like this:   Technically, it is possible to get to the page through selenium e.g. from selenium import webdriver driver = webdriver.Chrome("./lib/chromedriver_111.0.5563.64") driver.get('chrome://settings/help')  But even if we can get the selenium driver to get to the chrome://settings/help, the .page_source is information for Version ... is missing. Also, other than the chrome version, the access to chrome:// pages would be used to retrieve other information, e.g. https://www.ghacks.net/2012/09/04/list-of-chrome-urls-and-their-purpose/  While's there's a way to call a Windows command line function to retrieve the browser version details, e.g. How to get chrome version using command prompt in windows , the solution won't generalize and work on Mac OS / Linux.

Answer: You can use Pyppeteer as Selenium's alternative for accessing the chrome:// pages pip install pyppeteer asyncio  Example for Windows OS: from pyppeteer import launch import time import asyncio import nest_asyncio nest_asyncio.apply() async def main(): browser = await launch({ "headless": False, "executablePath": "C:/Program Files/Google/Chrome/Application/chrome.exe" }) page = await browser.newPage() await page.goto('chrome://settings/help') time.sleep(3) # wait 3s chromeVersion = await page.evaluate(''' document.querySelector('settings-ui') .shadowRoot.querySelector('settings-main') .shadowRoot.querySelector('settings-about-page') .shadowRoot.querySelector('div.secondary').innerText ''') await browser.close() print(chromeVersion) asyncio.get_event_loop().run_until_complete(main())  The output will correspond to your Chrome Application: Version 112.0.5615.87 (Official Build) (64-bit)  For Linux/Mac, you can change executablePath based on the Chrome application's path.
Question: I want the outline of a rectangle in a canvas to get a bigger width, when the rectangle is in state "disabled". Therefore I use the parameter "disabledwidth=4". But when the rectangle is in state "disabled", the outline has still a width of 1 instead of 4. This is my code, which shows the problem: When I move the mouse over the rectangle, the state of the rectangle changes to "active", everything works as expected, especially the outlinewidth changes to 4. But when I change the state to "disabled" by clicking on the button the outline stays at width 1. What am I doing wrong? import tkinter as tk def disabled(): canvas.itemconfig(rect, state="disabled") def normal(): canvas.itemconfig(rect, state="normal") root = tk.Tk() canvas = tk.Canvas(root, height=250, width=250) button1 = tk.Button(root, text="change rectangle to state disabled", command=disabled) button2 = tk.Button(root, text="change rectangle to state normal" , command=normal ) rect = canvas.create_rectangle(40, 40, 180, 180, fill = "red", activefill = "green2", activeoutline = "green3", activewidth = 4, disabledfill = "grey", disabledoutline= "grey2", disabledwidth = 4 ) canvas.grid() button1.grid() button2.grid() root.mainloop() 

Answer: This is due to a very old but simple bug in Tcl/Tk which causes -disabledwidth to not be parsed properly for rectangle and oval canvas items, causing it to be silently ignored (at least in some cases). I reported it to Tcl/Tk along with the fix: https://core.tcl-lang.org/tk/info/f4d9d74df628 So hopefully this will be fixed in Tcl/Tk 8.6.14 (which hopefully will then be used in future Python distributions).
Question: How can I add a column to a Polars DataFrame with current date and time as value on every row? With Pandas, I would do something like this: df["date"] = pd.Timestamp.today() 

Answer: EDIT: Better answer from @jqurious: df.with_columns(date = datetime.now())  My original solution: Use pl.lit() and Python's datetime to create a literal of the current date: from datetime import datetime import polars as pl df = pl.DataFrame( ... ) df.with_columns(pl.lit(datetime.now()).dt.datetime().alias("date"))
Question: I have a relplot with columns split on one variable. I'd like to add one additional column with no subplot or subpanel. To give a clear example, suppose I have the following plot: import matplotlib.pyplot as plt import numpy as np import pandas as pd import seaborn as sns x = np.random.random(10000) t = np.random.randint(low=0, high=3, size=10000) y = np.multiply(x, t) df = pd.DataFrame({'x': x, 't': t, 'y': y}) g = sns.relplot(df, x='x', y='y', col='t')  This generates a plot something like  I want a 4th column for t=3 that displays no data nor axes. I just want a blank white subplot of equal size as the first three subplots. How can I do this?

Answer: Add a value not observed in the data to col_order, e.g. g = sns.relplot(df, x='x', y='y', col='t', col_order=[0, 1, 2, ""]) g.axes.flat[-1].set_title("")
Question: I'm new to coding and struggling a bit with this one. Can anyone show me how to write the pseudocode for a program that finds the non-duplicate numbers and duplicate numbers (as simply as possible.) thanks! ** input:** 56 75 1 46 100 97 75 46 46 Output: Non-duplicates: 56 1 100 97 Duplicates: 75 46  I tried : list = [56, 75, 1, 46, 100, 97, 75, 46, 46] dupes = [] unique = [] for x in list: if x not in dupes: dupes.append(x) else: unique.append(x) print(dupes) print(unique)  However, I beleive there might be a more elegant solution (that matches the output.) The exercise I took this from was about sets. I'm not sure if that's too relevant! Thanks

Answer: Use count() method: lst = [56, 75, 1, 46, 100, 97, 75, 46, 46] dupes = [] unique = [] for x in lst: if lst.count(x) == 1: unique.append(x) elif x not in dupes: dupes.append(x) print('Non-duplicates:', *unique, sep='\n',) print() print('Duplicates:', *dupes, sep='\n')
Question: Sorry for the poor title but I'm unsure how better to describe the question. So I recently watched Andrej Kaparthy's build GPT video which is awesome and now trying to reconstruct the code myself I notices that he uses self() as a function and was curious why and what exactly it does. The code is here and I'm curious in particular about the generate function: class BigramLanguageModel(nn.Module): def __init__(self, vocab_size): super().__init__() # each token directly reads off the logits for the next token from a lookup table self.token_embedding_table = nn.Embedding(vocab_size, vocab_size) def forward(self, idx, targets=None): # idx and targets are both (B,T) tensor of integers logits = self.token_embedding_table(idx) # (B,T,C) if targets is None: loss = None else: B, T, C = logits.shape logits = logits.view(B*T, C) targets = targets.view(B*T) loss = F.cross_entropy(logits, targets) return logits, loss def generate(self, idx, max_new_tokens): # idx is (B, T) array of indices in the current context for _ in range(max_new_tokens): # get the predictions logits, loss = self(idx) # focus only on the last time step logits = logits[:, -1, :] # becomes (B, C) # apply softmax to get probabilities probs = F.softmax(logits, dim=-1) # (B, C) # sample from the distribution idx_next = torch.multinomial(probs, num_samples=1) # (B, 1) # append sampled index to the running sequence idx = torch.cat((idx, idx_next), dim=1) # (B, T+1) return idx  So to me it seems that he is calling the forward function defined within the class through using the self(). Is that correct? And if so why would he not use forward(idx) instead. Thank you for you help!

Answer: Meh, this is pytorch. Remember that you can use the model like this: model(x) to do the model.forward(x). So inside of the model class self(x) will be the basically the same as doing self.forward(x).
Question: I will show you some minimal example: from pydantic import BaseModel class Foo(BaseModel): value: PositiveInt | None = None def some_function(self) -> PositiveInt: return self.value  In above example there is problem with type of the returned value of "some_function". The "self.value" is not "PositiveInt" - its "Optional[PositiveInt, None]" - cause I want it during initialization as None - then doing some "_post_init_" (what is not supported right now) and return just "PositiveInt". What do you think, how should I do it? For example I can add "assert self.value" - in this case I will have assertion error when self.value is None. Or maybe just do "return PositiveInt(self.value)" - to prevent some mypy errors? Or maybe there is some internal build-in method for changing the type of returning value? (something like response_type in FastApi - when you need or want to return some data that is not exactly what the type declares) Any ideas? BR! Edit: More details: What if you want to have some checking if value1 or value2 exists and based on it calculate the second one? from pydantic import BaseModel class Foo(BaseModel): value1: PositiveInt | None = None value2: PositiveInt | None = None def post_init_function(self) -> None: if self.value1: self.value2 = 100 elif self.value2: self.value1 = 100 @property def value1(self) -> PositiveInt: return self.PositiveInt foo1 = Foo(value1=10) foo1.post_init_function() # value1 = 10, value2 = 100 foo2 = Foo(value2=5) foo2.post_init_function() # value1 = 100, value2 = 5 

Answer: Using dataclasses from the default library, I would separate the __init__ parameter that could be None from the field which cannot be None. from dataclasses import dataclass, field @dataclass class Foo: value1: int = field(init=False) value2: int = field(init=False) v1: InitVar[int | None] = None v2: InitVar[int | None] = None def __post_init__(self, v1, v2): if v1 is None and v2 is None: raise ValueError("v1 and v2 cannot both be None") elif v1 is None: self.value1 = 100 self.value2 = v2 else: self.value1 = v1 self.value2 = 100  I am not sure what the equivalent idiom for a Pydantic model would be, though I think this should work with few or no changes for a Pydantic dataclass.
Question: Not sure if it's off topic but don't know any better place to ask. In cpython there was a very giant switch case statement for executing each opcode. This switch case previously was placed in the _PyEval_EvalFrameDefault function. Here is the link. The switch case statement starts here. This was a core part of cpython and everyone who was interested in cpython internals, would probably explore it in detail. Recently I was searching for it and I couldn't find it. In this version of _PyEval_EvalFrameDefault I can't find it. It's much shorter than the last one. I even tried to find this switch statement by searching for opcodes in my IDE. But even that didn't helped find where it is. Can anyone who is aware of latest cpython development changes help me? Thanks in advance.

Answer: It's in Python/generated_cases.c.h, which gets inserted into _PyEval_EvalFrameDefault with an #include "generated_cases.c.h". As you might guess from the name, generated_cases.c.h is generated code. You can see the code generator in Tools/cases_generator/generate_cases.py
Question: Previously I had a pandas dataframe that I could display as a table in Databricks using: df.display()  Pandas was updated to v2.0.0. today and I am now getting the following error when I run df.display(): AttributeError: 'DataFrame' object has no attribute 'iteritems'  Anyone know how I can resolve this? I tried running df.display (without parenthesis) and it gives an output but I am looking for an output in the tabular form.

Answer: As a workaround, downgrade to pandas v1.5 %pip install --upgrade pandas==1.5   The answers provided till now used to work prior to 3rd April 2023. As of April 4, with pandas 2.0.0, you are not able to convert a Pandas DataFrame to a Spark DataFrame using the command: spark.createDataFrame(df)  Using the above command leads to the error mentioned in the question: AttributeError: 'DataFrame' object has no attribute 'iteritems'  The iteritems function seems to have been removed in pandas 2.0.0. From the changelog of pandas 2.0.0: Removed deprecated Series.iteritems(), DataFrame.iteritems(), use obj.items instead  While the code written in spark to convert pandas dataframe to a spark dataframe still uses iteritems /databricks/spark/python/pyspark/sql/pandas/conversion.py in createDataFrame(self, data, schema, samplingRatio, verifySchema) 308 warnings.warn(msg) 309 raise --> 310 data = self._convert_from_pandas(data, schema, timezone) 311 return self._create_dataframe(data, schema, samplingRatio, verifySchema) 312 /databricks/spark/python/pyspark/sql/pandas/conversion.py in _convert_from_pandas(self, pdf, schema, timezone) 340 pdf[field.name] = s 341 else: --> 342 for column, series in pdf.iteritems(): 343 s = _check_series_convert_timestamps_tz_local(series, timezone) 344 if s is not series:  Looks like we will have to wait for a fix to use Pandas 2.0.0.
Question: I have a pandas data frame which I want to convert into spark data frame. Usually, I use the below code to create spark data frame from pandas but all of sudden I started to get the below error, I am aware that pandas has removed iteritems() but my current pandas version is 2.0.0 and also I tried to install lesser version and tried to created spark df but I still get the same error. The error invokes inside the spark function. What is the solution for this? which pandas version should I install in order to create spark df. I also tried to change the runtime of cluster databricks and tried re running but I still get the same error. import pandas as pd spark.createDataFrame(pd.DataFrame({'i':[1,2,3],'j':[1,2,3]})) error:- UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true; however, failed by the reason below: 'DataFrame' object has no attribute 'iteritems' Attempting non-optimization as 'spark.sql.execution.arrow.pyspark.fallback.enabled' is set to true. warn(msg) AttributeError: 'DataFrame' object has no attribute 'iteritems' 

Answer: It's related to the Databricks Runtime (DBR) version used - the Spark versions in up to DBR 12.2 rely on .iteritems function to construct a Spark DataFrame from Pandas DataFrame. This issue was fixed in the Spark 3.4 that is available as DBR 13.x. If you can't upgrade to DBR 13.x, then you need to downgrade the Pandas to latest 1.x version (1.5.3 right now) by using %pip install -U pandas==1.5.3 command in your notebook. Although it's just better to use Pandas version shipped with your DBR - it was tested for compatibility with other packages in DBR.
Question: We have two different libraries, one in Python and one in Go that need to compute murmur3 hashes identically. Unfortunately no matter how hard we try, we cannot get the libraries to produce the same result. It appears from this SO question about Java and Python that compatibility isn't necessarily straight forward. Right now we're using the python mmh3 and Go github.com/spaolacci/murmur3 libraries. In Go: hash := murmur3.New128() hash.Write([]byte("chocolate-covered-espresso-beans")) fmt.Println(base64.RawURLEncoding.EncodeToString(hash.Sum(nil))) // Output: cLHSo2nCBxyOezviLM5gwg  In Python: name = "chocolate-covered-espresso-beans" hash = mmh3.hash128(name.encode('utf-8'), signed=False).to_bytes(16, byteorder='big', signed=False) print(base64.urlsafe_b64encode(hash).decode('utf-8').strip("=")) # Output: jns74izOYMJwsdKjacIHHA (big byteorder) hash = mmh3.hash128(name.encode('utf-8'), signed=False).to_bytes(16, byteorder='little', signed=False) print(base64.urlsafe_b64encode(hash).decode('utf-8').strip("=")) # Output: HAfCaaPSsXDCYM4s4jt7jg (little byteorder) hash = mmh3.hash_bytes(name.encode('utf-8')) print(base64.urlsafe_b64encode(hash).decode('utf-8').strip("=")) # Output: HAfCaaPSsXDCYM4s4jt7jg  In Go, murmur3 returns a uint64 so we assume signed=False in Python; however we also tried signed=True and did not get matching hashes. We're open to different libraries, but are wondering if there is something wrong with either our Go or Python methodologies of computing a base64 encoded hash from a string. Any help appreciated.

Answer: That first Python result is almost right. >>> binascii.hexlify(base64.b64decode('jns74izOYMJwsdKjacIHHA==')) b'8e7b3be22cce60c270b1d2a369c2071c'  In Go:  x, y := murmur3.Sum128([]byte("chocolate-covered-espresso-beans")) fmt.Printf("%x %x\n", x, y)  Results in: 70b1d2a369c2071c 8e7b3be22cce60c2  So the order of the two words is flipped. To get the same result in Python, you can try something like: name = "chocolate-covered-espresso-beans" hash = mmh3.hash128(name.encode('utf-8'), signed=False).to_bytes(16, byteorder='big', signed=False) hash = hash[8:] + hash[:8] print(base64.urlsafe_b64encode(hash).decode('utf-8').strip("=")) # cLHSo2nCBxyOezviLM5gwg
Question: How to interactively (micro)benchmark a code expression in Python (i.e. the equivalent of @btime/@benchmark in Julia) ? I would like to benchmark interactively an expression (a function call), where the number of times this expression should be evaluated depends on its computational cost / variance.... I have tried timeit.timeit but I have some quesitons:  How do I interpret the time output of the following results:  >>> a = 2.2 >>> timeit.repeat("round(a)", setup='from __main__ import a; gc.enable()', repeat=5,number=10) [4.631001502275467e-06, 1.3809185475111008e-06, 1.2170057743787766e-06, 1.1718366295099258e-06, 1.1730007827281952e-06] >>> timeit.timeit("round(a)", setup='from __main__ import a; gc.enable()', number=10) 5.741836503148079e-06 >>> timeit.timeit("round(a)", setup='from __main__ import a; gc.enable()') 0.11461802502162755 >>> timeit.Timer('round(a)', setup='from __main__ import a; gc.enable()').autorange() (5000000, 0.4272152939811349)  It seems that when I go from number from 1 to 10, the timing remains pretty stable, but then it grows of different order of magnitude....  Does it makes sense a workflow where I first create the Timer object once, and then I modify several times the code to evaluate and benchmark the modified Timer ? E.g.  import timeit import gc bclass = timeit.Timer('round(1.1)','gc.enable()', globals=globals()) [...] bclass.stmt = "small_3x2_nash.vertex_enumeration()" bclass.autorange() bclass.stmt = "small_3x2_nash.lemke_howson_enumeration()" bclass.autorange() bclass.stmt = "small_3x2_nash.support_enumeration()" bclass.autorange()  (seems this is not possible, as I have tried very different code and the result of the timing remains almost the same)

Answer: You can also use the Timer class to enable de GC, as explained in the docs here: https://docs.python.org/3/library/timeit.html#timeit.Timer.timeit And instead of explicit import every variable/function that you want to use from main, you could pass globals() in the Timer arguments. In that way, all the globals variables, functions, etc will be passed to de evaluated function. Python 3.10.10 (main, Mar 5 2023, 22:26:53) [GCC 12.2.1 20230201] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import timeit >>> >>> a = 2.2 >>> >>> t = timeit.Timer('round(a)', globals=globals()) >>> >>> t.timeit(number=1000) 0.00042562399994494626 >>> t.autorange() (5000000, 0.46861540100007915)  To enable GC you need also import the gclib import gc t = timeit.Timer('round(a)', 'gc.enable()', globals=globals())
Question: Here is a .pre-commit-config.yaml from pre-commit. It will git clone the specified rev of git repo. How can I ignore the rev and always git clone the newest? repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v2.3.0 hooks: - id: check-yaml - id: end-of-file-fixer - id: trailing-whitespace 

Answer: you intentionally cannot from the docs  pre-commit configuration aims to give a repeatable and fast experience and therefore intentionally doesn't provide facilities for "unpinned latest version" for hook repositories. Instead, pre-commit provides tools to make it easy to upgrade to the latest versions with pre-commit autoupdate. If you need the absolute latest version of a hook (instead of the latest tagged version), pass the --bleeding-edge parameter to autoupdate. pre-commit assumes that the value of rev is an immutable ref (such as a tag or SHA) and will cache based on that. Using a branch name (or HEAD) for the value of rev is not supported and will only represent the state of that mutable ref at the time of hook installation (and will NOT update automatically).   disclaimer: I wrote pre-commit