eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to run PyScript in a webpage, where I intended it to run and produce the output of a python code to a div element. I've succeeded in doing this. However I noticed it always produces the splashscreen "Downloading Pyodide Python Startup" every time I refreshed the webpage. It's taking significantly longer wait time for this. Is there a way to in a way cache the Pyodide installation and make the loading times significantly faster? I'm deploying this webpage to Github Pages here https://reza-nugraha32.github.io/geophysics-wiki/ . I'm aware that it's not better to hide the splash screen entirely. Is there also a way to start this splash screen in a div container or importing the PyScript only when a "Run" button is clicked? For instance, Educative.io here made it so it display "Loading" only when the "Run" button is clicked. I've tried deleting the defer tag from the PyScript import script but it gave me errors.

Answer: Sorry, Reza, but that's just the way it works. The whole environment has to be downloaded before any code runs. Pyscript Next (2023.11.1) may improve your situation, especially if you can use Micropython. Check out Jeff Glass's blog - https://jeff.glass/post/whats-new-pyscript-2023-11-1/.
Question: Locally the functions debug just fine, if I deploy via vscode to my azure function I get No HTTP Triggers found and the devops pipeline does not deploy triggers either. I have "AzureWebJobsFeatureFlags": "EnableWorkerIndexing" set locally and as a function app setting. Code is appropriately decorated @app.route(route="functionname", auth_level=func.AuthLevel.FUNCTION) def functioname(req: func.HttpRequest) -> func.HttpResponse:  Deployments succeed both ways but no functions show Azure Pipeline shows correct files:  Azure function app files show function_app.py at the root folder test function app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS) @app.function_name("personas") @app.route(route="character-managment/personas") def personas(req: func.HttpRequest) -> func.HttpResponse: logging.info('Python HTTP trigger function processed a request.') return func.HttpResponse("ok", status_code=200)  Folder structure  Works locally 

Answer: I tried to reproduce the same in my environment. I could see the deployed function in Azure function app.  Created a test default Python v2 function and deployed to Azure function app.  Local:   Make sure you added the setting "AzureWebJobsFeatureFlags": "EnableWorkerIndexing" in local.settings.json:  { "IsEncrypted": false, "Values": { "FUNCTIONS_WORKER_RUNTIME": "python", "AzureWebJobsFeatureFlags": "EnableWorkerIndexing", "AzureWebJobsStorage": "<storage_connection_string>" } }   Create and activate a new virtual environment using the commands:  py -m venv .venv .\.venv\Scripts\activate    Include all the required packages in requirements.txt file. Install all the packages by running the command pip install -r requirements.txt in the terminal.    Created a Python function app with Consumption plan and version 3.10.  Deployed the function with the command func azure functionapp publish <function_app_name>:     Portal:    I could also deploy the function with custom packages to Azure Function App. requirements.txt: azure-functions numpy pandas  Code(function_app.py): import azure.functions as func import logging import numpy as np import pandas as pd app = func.FunctionApp() @app.route(route="HttpTrigger", auth_level=func.AuthLevel.ANONYMOUS) def HttpTrigger(req: func.HttpRequest) -> func.HttpResponse: logging.info('Python HTTP trigger function processed a request.') arr = np.array([1, 2, 3]) df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}) name = req.params.get('name') if not name: try: req_body = req.get_json() except ValueError: pass else: name = req_body.get('name') if name: return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully with {arr} and {df}.") else: return func.HttpResponse( "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.", status_code=200 )  Local:    Portal:   References: Create a Python function from the command line You can also refer GitHub ticket raised on the same issue.
Question: I have a project where I am trying to finetune Llama-2-7b on a dataset for Parameter extraction, which is linked here: <GalaktischeGurke/parameter_extraction_1500_mail_contract_invoice>. The problem with the dataset is that the context for a response is very big, meaning that training on the entire dataset with context, not only on the response results in a huge loss of performance. To fix this issue, I wanted to use SFT_trainer together with the DataCollatorForCompletionOnlyLM, which allows finetuning only for response. Now, before adjusting my training loop, I wanted to try the examples given here: https://huggingface.co/docs/trl/main/en/sft_trainer. Specifically, I used this code from the page: from transformers import AutoModelForCausalLM, AutoTokenizer from datasets import load_dataset from trl import SFTTrainer, DataCollatorForCompletionOnlyLM dataset = load_dataset("timdettmers/openassistant-guanaco", split="train") output_dir = "./results" model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m") tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") instruction_template = "### Human:" response_template = "### Assistant:" collator = DataCollatorForCompletionOnlyLM(instruction_template=instruction_template, response_template=response_template, tokenizer=tokenizer, mlm=False) trainer = SFTTrainer( model, train_dataset=dataset, dataset_text_field="text", data_collator=collator, ) trainer.train() import os output_dir = os.path.join(output_dir, "final_checkpoint") trainer.model.save_pretrained(output_dir)  The training loop did not crash, but it never seemed to train at all - There was no train/loss curve on wandb and the model saved didnt seem to have changed. These are the things I tried: -Using the other code with preformat function -setting packing=False on the trainer -implementing it with my own loop, which yielded the same results -trying to find documentation on the collator, however it is not in the official docs at https://huggingface.co/docs/transformers/main_classes/data_collator Does anyone know what the issue is here?

Answer: I have a similar issue. I think you're forgetting to add formatting_func function. Also, by default setting dataset_text_field overrides the use of the collator, so try without that argument. Here's how I call it. It runs and stores things to wandb, but my problem is my loss is always NaN. Lemme know if you found the issue! trainer = SFTTrainer( model, train_dataset=vanilla_data_set, eval_dataset=vanilla_data_set, args=training_args, # dataset_text_field="gpt-4", # torch_dtype=torch.bfloat16, peft_config=peft_config, max_seq_length=512, formatting_func=formatting_prompts_func, data_collator=collator )
Question: I'm using Pycharm to develop an app with SQLAlchemy 2.0. When I attempt to query some table using ORM approach. Pycharm always display type error in the filter query. For example, in the code snippet below:  with Session(engine) as session: session.scalars(select(Albums.AlbumId).where(Albums.Id > user_last_sync_id)) ^^^ Show wrong type  Get the following message Expected type 'ColumnElement[bool] | _HasClauseElement | SQLCoreOperations[bool] | ExpressionElementRole[bool] | () -> ColumnElement[bool] | LambdaElement'，, got 'bool' instead   Even though it indicates a type error, but scripts still be executed (And get correct data) without showing any error messages. What could potentially be causing this issue in the code? Is there a way to make code more "correct" to let Pycharm not to display type error?

Answer: PyCharm assumes that expressions of the form a > b evaluate to bool when no other type information is available. Most likely, SQLAlchemy isn't providing rich enough type hints and/or stubfiles for PyCharm to correctly infer the type of that expression. To resolve that warning, you can inform PyCharm about the true type of the expression using typing.cast: .where( cast("ColumnElement[bool]", Albums.Id > user_last_sync_id) )  Alternatively, you can suppress the warning by adding # type: ignore to the end of the line.
Question: I have a pandas DataFrame with the columns ["x", "y", "r", "g", "b"] where x and y denote the coordinates of a pixel and r, g, b denote its RGB value. The rows contain entries for each coordinate of a grid of pixels and are unique. How can I display this DataFrame using matplotlibs's imshow()? This requires reshaping the data into a array of shape (M, N, 3). My usual approach of using plt.imshow(df.pivot(columns="x", index="y", values="i"), interpolation="nearest") does only work for greyscale images. Placing ["r", "g", "b"] as the values argument yields a DataFrame with a MultiIndex as columns. However I fail to convert this into a correct image. Simply calling .reshape(M, N, 3) creates a wrong image. I also had the idea of creating a new column with df["rgb"] = list(zip(df.r, df.g, df.b)) However I'm not sure on how to convert the resulting tuples into a new axis for the ndarray.

Answer: There exists an easy way to do this. First, you make sure the DataFrame is sorted by x- and y-values using df = df.sort_values(by=['x', 'y']). Next, you select only the three columns for r, g and b from the DataFrame by calling df[['r', 'g', 'b']]. You convert the values into a numpy array by calling df[['r', 'g', 'b']].values, which will return an array of the shape (M*N, 3), assuming that M and N are the width and height of your image. Now, reshape that array into the shape (M, N, 3) and you are done. df = df.sort_values(by=['x', 'y']) values = df[['r', 'g', 'b']].values image = values.reshape(df['x'].max() + 1 , df['y'].max() + 1, 3)  I'm assuming here that your x and y values in the DataFrame start at 0, therefore I add 1 for the dimensions. If your x and y values start at 1, the reshaping can be done like this (df['x'].max(), df['y'].max(), 3). Depending on what you consider the x and y dimensions of your image, you might have to transpose the array in the end.
Question: I need to append the first item of a (general) iterable as the final item of that iterable (thus "closing the loop"). I've come up with the following: from collections.abc import Iterable from itertools import chain def close_loop(iterable: Iterable) -> Iterable: iterator = iter(iterable) first = next(iterator) return chain([first], iterator, [first]) # Examples: list(close_loop(range(5))) # [0, 1, 2, 3, 4, 0] ''.join(close_loop('abc')) # 'abca'  Which works, but seems a bit "clumsy". I was wondering if there's a more straightforward approach using the magic of itertools. Solutions using more_itertools are also highly welcome.

Answer: Here is a version which doesn't use itertools def close_loop(iterable): iterator = iter(iterable) first = next(iterator) yield first yield from iterator yield first
Question: How could I see/log queries sent by sqlmodel to database.

Answer: Since sqlmodel uses Sqlalchemy as backend ORM engine, we can use Sqlalchemy logging answer like answer here: import logging logging.basicConfig() logger = logging.getLogger('sqlalchemy.engine') logger.setLevel(logging.DEBUG) # run sqlmodel code after this  You should be able to see sql queries in the console.
Question: Assume we have an f-string-style template: "{kid} ate {number} {fruit}" and a formatted version of that: "Jack ate 42 apples" How could I, very generally, extract "Jack", "42", and "apples" from this string based on the fact that they match kid, number, and fruit respectively? Assume that the f-string could be anything and contain any number of simple string or integer variables (no formatting instructions), and that the string for extraction has already been generated from that f-string, and so matches it perfectly. This also means that I can't use regex rather than f-string-style formatting, unless there's a way to generate a regex based on the f-string. I already have something that works for my case, but it's not general, and feels a little hacky to me. I feel like there must be something simpler that I just can't find the right words to google: template = "{kid} ate {number} {fruit}" example = "Jack ate 42 apples" extracted = {"kid": "", "number": "", "fruit": ""} for variable in extracted.keys(): extracted[variable] = example for chunk in template.split(f"{{variable}}"): extracted[variable] = extracted[variable].replace(chunk, "")  If it's not possible to get both more simple and more general, then more simple with the same level of generality would be great too :)

Answer: You can do this with regex and match group like this:  To create a group with name: (?'name') Match any characters unlimited times: .+  import re pattern = r"(?P<name>.+) ate (?P<number>.+) (?P<fruit>.+)" text = "John ate 3 apples" match = re.match(pattern, text) if match: name = match.group('name') number = match.group('number') fruit = match.group('fruit') print(f"{name} ate {number} {fruit}") else: print("Pattern not found in the text.")  I hope this is what you want
Question: I'm trying to make a text based game and I want the text to print out slowly to simulate typing. I also want it to be an input. Is it possible to do this? I saw someone else use this code(bolded is what i added): import sys,time def sinput(str): for c in str + '\n': sys.stdout.write(c) sys.stdout.flush() time.sleep(4./90) slowtest=sinput('Does it work? ') **if slowtest == 'yes': print('Amazing!')**  And it was able to print out the code slowly just like I wanted, unfortunately I can not answer the question ("Does it work?").

Answer: You can assign input() to slowtest. sinput() function is not returning anything. Once you print slowly you can use input() for slowtest variable then check the if condition. import sys,time def sinput(str): for c in str + '\n': sys.stdout.write(c) sys.stdout.flush() time.sleep(4./90) sinput('Does it work? ') slowtest = input() if slowtest == 'yes': print('Amazing!')
Question: I'm trying to broadcast the contents of one array into another array like this: A = np.array([[1, 3], [2, 4]]) A_broadcast = np.array([[1, 0, 3, 0], [0, 2, 0, 4], [1, 2, 3, 4]])  My current approach is by initializing A_broadcast with np.zeros((3, 4)) and slicing the contents of A into A_broadcast one line at a time like this: A_broadcast[::2][0] = A[0] A_broadcast[1::2][1] = A[1] A_broadcast[::2][2] = A[0] A_broadcast[1::2][2] = A[1]  But I get this error: ValueError: could not broadcast input array from shape (2,) into shape (4,) This approach works in Matlab so I thought something similar would work here. Is there a way this approach can work? If not, what could I do to get a similar effect?

Answer: Your 2 arrays: In [86]: A = np.array([[1, 3], [2, 4]]) ...: A_broadcast = np.array([[1, 0, 3, 0], [0, 2, 0, 4], [1, 2, 3, 4]]) In [88]: A Out[88]: array([[1, 3], [2, 4]]) In [89]: A_broadcast Out[89]: array([[1, 0, 3, 0], [0, 2, 0, 4], [1, 2, 3, 4]])  the blank: In [87]: res = np.zeros((3,4),int)  The first row of A goes into a square that alternates in both directions: In [90]: res[::2,::2]=A[0] In [91]: res Out[91]: array([[1, 0, 3, 0], [0, 0, 0, 0], [1, 0, 3, 0]])  Actually I wasn't sure if A[0] was right its (2,1) version. I could have analzed the case, but instead just tried it. Sometimes in an interactive session it's easier to try a few alternatives than to carefully work out the details in my head before hand. It's easy to experient in an interactive session with modules like numpy. Now it's easy to do the same with the 2nd row, In [92]: res[1:,1::2]=A[1] In [93]: res Out[93]: array([[1, 0, 3, 0], [0, 2, 0, 4], [1, 2, 3, 4]])  Nothing wrong with indexing the target one row at a time as you attempt: In [94]: res[0,::2] Out[94]: array([1, 3]) In [95]: res[2,1::2] Out[95]: array([2, 4])  Both rows could also be indexed with advanced indexing - we just have to get the broadcasting right, using a (2,1), and (2,) arrays (or list equivalents): In [96]: res[[[0],[2]],[0,2]] Out[96]: array([[1, 3], [1, 3]]) In [97]: res[[[1],[2]],[1,3]] Out[97]: array([[2, 4], [2, 4]])
Question: I access a function via this list comprehension and have made it work by explicitly creating a variable for each element of the list_of_lists. I want a better way to access the elements in the function in a list comprehension. Example: list_of_lists = [[0, 1, 2, 3], [0, 1, 2, 3], ...] [function(i, j, k, l) for i, j, k, l in list_of_lists]  It's a very annoying syntax as I need to update (i, j, k, l) for i, j, k, l if the number of elements in a sub-list of list_of_lists changes. E.g., a change to [[0, 1, 2, 3, 4, 5, 6], ...] needs the syntax to be (i, j, k, l, m, n) for i, j, k, l, m, n and I need to be sure I do not miscount. This gets worse for more elements per sub-list and if the function changes during coding. Is there a way to say something like: [function(*) for * in list_of_lists]  So my woes are ameliorated? I tried searching for something like this but it's clear I don't know the right words to be able to search this.

Answer: You're looking for * to unpack the list https://peps.python.org/pep-3132/ [function(*sublist) for sublist in list_of_lists]  >>> def foo(a, b, c, d): ... return a + b + c + d ... >>> lst = [1, 2, 3, 4] >>> foo(*lst) # iterable unpack 10 >>> d = {'a':1, 'b':2, 'c':3, 'd':4} >>> foo(**d) # dict unpack 10
Question: I was surprised to read here that  The start and step arguments default to None  since it also says:  slice(start, stop, step=1)  Return a slice object representing the set of indices specified by range(start, stop, step).  So I expected the default argument value for the step parameter to be 1. I know that slice(a, b, None) == slice(a, b, 1) returns False, but I am curious if slice(a, b, None) always returns the same slice as slice(a, b, 1), or if there is some example that I haven't been able to think of for which they will return different slices. I couldn't find anything about this in the extensive post on slicing here

Answer: Slice's step indeed defaults to None, but using step 1 and None should be equivalent for all practical purposes. That's because in the C code where the step is actually used, there are checks which transform None into 1 anyway: int PySlice_GetIndices(PyObject *_r, Py_ssize_t length, Py_ssize_t *start, Py_ssize_t *stop, Py_ssize_t *step) { PySliceObject *r = (PySliceObject*)_r; if (r->step == Py_None) { *step = 1; } ... }  And: int PySlice_Unpack(PyObject *_r, Py_ssize_t *start, Py_ssize_t *stop, Py_ssize_t *step) { PySliceObject *r = (PySliceObject*)_r; ... if (r->step == Py_None) { *step = 1; } ... }  If you're wondering why they don't just default to 1 instead, perhaps it's because users may still want to slice using None explicitly (e.g. L[1:2:None]), and/or to give 3rd-party types the opportunity to handle 1 and None differently in their __getitem__/__setitem__/__delitem__ implementation.
Question: Suppose I have a tree-like data structure in Python: class Node: def __init__(self, name: str, neighbors: Optional[Iterable[Node]] = None) -> None: self.name = name self.neghbors = neighbors or [] grand_child1 = Node("grand_child1") grand_child2 = Node("grand_child1") child = Node("child", [grand_child1, grand_child2]) root = Node("root", [child])  Now I want to render this structure into a ul in HTML. I want the output to be something like this: <ul> <li>root</li> <ul> <li>child</li> <ul> <li>grand_child1</li> <li>grand_child2</li> </ul> <ul> </ul>  To do this I want to use a Jinja template. The what I want to extract from each node is the same, it's just that I want to capture the tree structure and avoid duplicate code. Is there a way to use a template recursively? Something like this: <ul> {% for root in roots %} <li>{{ root.name }}</li> {# this template applied for the neighbors #} {% endfor %} </ul> 

Answer: Turns out you can define macros in jinja which work like functions. Something like this will do the trick: {% macro make_ul(roots) -%} <ul> {% for root in roots %} <li>{{ root.name }}</li> {{ make_ul(root.neighbors) }} {% endfor %} </ul> {%- endmacro %} {{ make_ul(roots) }}  This way the macro will call itself recursively till it hits the leaves of the tree.
Question: I am currently working on automating the login process for my Twitter account using Python and Selenium. However, I'm facing an issue where Twitter's anti-bot measures seem to detect the automation and immediately redirect me to the homepage when clicking the next button.  I have attempted to use send_keys and ActionChains to create more human-like interactions, but the problem persists. Here's a simplified code snippet that illustrates my current approach: # imports... driver.get(URLS.login) username_input = driver.find_element(By.NAME, 'text') username_input.send_keys(username) next_button = driver.find_element(By.XPATH, '//div[@role="button"]') # These attempts all failed and return to the homepage next_button.click() next_button.send_keys(Keys.ENTER) ActionChains(driver).move_to_element(next_button).click().perform()  What's weird is that besides manually clicking the next button, execute a click in console also works. I suspect that my automation attempts are still being detected by Twitter's security mechanisms, but I'm unsure about the root cause or how to bypass it successfully.

Answer: You may try this to log in to Twitter: import time from selenium import webdriver from selenium.webdriver import ChromeOptions, Keys from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.wait import WebDriverWait options = ChromeOptions() options.add_argument("--start-maximized") options.add_experimental_option("excludeSwitches", ["enable-automation"]) driver = webdriver.Chrome(options=options) url = "https://twitter.com/i/flow/login" driver.get(url) username = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'input[autocomplete="username"]'))) username.send_keys("your_username") username.send_keys(Keys.ENTER) password = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'input[name="password"]'))) password.send_keys("your_password") password.send_keys(Keys.ENTER) time.sleep(10)  reference:  https://github.com/ajeet214/Web_Scraping_with_Selenium/blob/main/twitter_login.py
Question: I would like to fit a polynomial to a series of x/y-datapoints and evaluate it at arbitrary x-values. I am using the new numpy polynomial API: import matplotlib.pyplot as plt import numpy as np x = [50.0, 150.0, 250.0, 400.0, 600.0, 850.0, 1000.0] y = [3.2, 10.1, 16.3, 43.7, 69.1, 45.2, 10.8] mypol = np.polynomial.polynomial.Polynomial.fit( x = x, y = y, deg = 3, ) np.polyval(mypol.coef, x)  Unfortunately, this returns values which are far greater than the original y-values: array([7.34024294e+06, 1.96122504e+08, 9.06027339e+08, 3.70658027e+09, 1.25012342e+10, 3.55289563e+10, 5.78446450e+10])  I played around with the order of the polynomial, but wasn't able to get anything useful from the function. What am I doing wrong? 

Answer: Numpy's polyval should be evaluated at the x-points, not the y-points. On top of that, the coefficients should be given in order of highest to lowest degree. That said, mypol.coef does not necessarily return the expanded polynomial coefficients, so mypol.coef may be useless for np.polyval. Instead, the numpy.polynomial.polynomial.Polynomial object returned by numpy.polynomial.Polynomial.fit can be evaluated at the x-points of interest, i.e. you can do mypol(x). Just note that x here should be a numpy array. Also note that the fit polynomial is a least-squares fit, meaning it is not an exact interpolation. import matplotlib.pyplot as plt import numpy as np from numpy.polynomial import Polynomial x = np.array([50.0, 150.0, 250.0, 400.0, 600.0, 850.0, 1000.0]) y = np.array([3.2, 10.1, 16.3, 43.7, 69.1, 45.2, 10.8]) mypol = Polynomial.fit(x, y, 3) x_fine = np.linspace(x.min(), x.max(), 100) y_fine = mypol(x_fine) fig, ax = plt.subplots() ax.scatter(x, y, label="data") ax.plot(x_fine, y_fine, "tab:orange", label="fit") ax.set(xlabel="x", ylabel="y", title="NumPy Polynomial Fit") ax.legend()