eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to run a python script from Jenkins using Jenkinsfile. Is there any way to run it directly from Jenkinsfile.  I found python plugin(Click Here) in Jenkins to run a script, but there is no proper documentation for this plugin. It would be very helpful if anyone explains how to integrate this plugin with Jenkinsfile.

Answer:  Adds the ability to execute python scripts as build steps. Other than that, this plugin works pretty much like the standard shell script support  Per the docs of the plugin. Though I have not used this plugin through pipeline, from job perspective, you have to just provide .py script (filename and path), in a same way you provide for shell/powershell script. Similarly, even for python, you'll be executing the script on a node, which will be either Linux or Windows. So, it would work as below : stage('build') { steps { sh 'python abc.py' } }  References : https://www.jenkins.io/doc/pipeline/tour/hello-world/ Lookout for "Python" block.
Question: I'm plotting time series data using seaborn lineplot (https://seaborn.pydata.org/generated/seaborn.lineplot.html), and plotting the median instead of mean. Example code: import seaborn as sns; sns.set() import matplotlib.pyplot as plt fmri = sns.load_dataset("fmri") ax = sns.lineplot(x="timepoint", y="signal", estimator = np.median, data=fmri)  I want the error bands to show the interquartile range as opposed to the confidence interval. I know I can use ci = "sd" for standard deviation, but is there a simple way to add the IQR instead? I cannot figure it out. Thank you!

Answer: I don't know if this can be done with seaborn alone, but here's one way to do it with matplotlib, keeping the seaborn style. The describe() method conveniently provides summary statistics for a DataFrame, among them the quartiles, which we can use to plot the medians with inter-quartile-ranges. import seaborn as sns; sns.set() import matplotlib.pyplot as plt fmri = sns.load_dataset("fmri") fmri_stats = fmri.groupby(['timepoint']).describe() x = fmri_stats.index medians = fmri_stats[('signal', '50%')] medians.name = 'signal' quartiles1 = fmri_stats[('signal', '25%')] quartiles3 = fmri_stats[('signal', '75%')] ax = sns.lineplot(x, medians) ax.fill_between(x, quartiles1, quartiles3, alpha=0.3);
Question: I am using discord.py to make a bot, and there are more commands than can fit on one page for my custom help command. I want the bot to add 2 reactions, back and forward, then the user that sent the help message can pick one, and go onto different pages of the help command. I want the bot to be able to edit the message to show the second page, and if they go back, then edit back to the original first page. Could anyone help with this? This is similar to the owobot definitions, where you can scroll back and forth between definitions.

Answer: This method would be using Client.wait_For(), and can be easily adapted if you have any other ideas for it. Example @bot.command() async def pages(ctx): contents = ["This is page 1!", "This is page 2!", "This is page 3!", "This is page 4!"] pages = 4 cur_page = 1 message = await ctx.send(f"Page {cur_page}/{pages}:\n{contents[cur_page-1]}") # getting the message object for editing and reacting await message.add_reaction("◀️") await message.add_reaction("▶️") def check(reaction, user): return user == ctx.author and str(reaction.emoji) in ["◀️", "▶️"] # This makes sure nobody except the command sender can interact with the "menu" while True: try: reaction, user = await bot.wait_for("reaction_add", timeout=60, check=check) # waiting for a reaction to be added - times out after x seconds, 60 in this # example if str(reaction.emoji) == "▶️" and cur_page != pages: cur_page += 1 await message.edit(content=f"Page {cur_page}/{pages}:\n{contents[cur_page-1]}") await message.remove_reaction(reaction, user) elif str(reaction.emoji) == "◀️" and cur_page > 1: cur_page -= 1 await message.edit(content=f"Page {cur_page}/{pages}:\n{contents[cur_page-1]}") await message.remove_reaction(reaction, user) else: await message.remove_reaction(reaction, user) # removes reactions if the user tries to go forward on the last page or # backwards on the first page except asyncio.TimeoutError: await message.delete() break # ending the loop if user doesn't react after x seconds  If your editor doesn't support pasting in the emojis directly, you can use a website such as this one to find the unicodes of the emojis instead. In this case, the forwards arrow was \u25c0 and the backwards arrow was \u25b6. Other than that, you should be good to go! The message will delete itself after 60 seconds of inactivity in that message (i.e. nobody reacting with the arrows), but just change the number if you want a longer period before deletion. Alternatively, you can add in a third emoji, such as a cross, which which delete the message on demand.  References:  Message.add_reaction() Message.remove_reaction() Client.wait_for() Message.edit() Message.delete() asyncio.TimeoutError - Exception for when user doesn't react in time
Question: I use tox to test a python project with the following basic config (tox.ini): [tox] envlist = py3 isolated_build = True [testenv] deps = pytest pytest-cov commands = pytest --cov {envsitepackagesdir}/foobar --cov-report xml --cov-report term  Unfortunately, the package's optional dependencies (as specified in setup.cfg) don't get installed; the corresponding line in raw pip would be pip install .[all]  How to make tox install all optional dependencies?

Answer: The supported way to do this is to use the extras key in your testenv for example: [testenv] deps = -rrequirements-dev.txt extras = typed  this will install .[typed] or -e .[typed] if usedevelop = true  disclaimer: I'm one of the tox maintainers
Question: I've been using FastAPI to create an HTTP based API. It currently supports JSON encoded parameters, but I'd also like to support form-urlencoded (and ideally even form-data) parameters at the same URL. Following on Nikita's answer I can get separate urls working with: from typing import Optional from fastapi import FastAPI, Body, Form, Depends from pydantic import BaseModel class MyItem(BaseModel): id: Optional[int] = None txt: str @classmethod def as_form(cls, id: Optional[int] = Form(None), txt: str = Form(...)) -> 'MyItem': return cls(id=id, txt=txt) app = FastAPI() @app.post("/form") async def form_endpoint(item: MyItem = Depends(MyItem.as_form)): print("got item =", repr(item)) return "ok" @app.post("/json") async def json_endpoint(item: MyItem = Body(...)): print("got item =", repr(item)) return "ok"  and I can test these using curl by doing: curl -X POST "http://localhost:8000/form" -d 'txt=test'  and curl -sS -X POST "http://localhost:8000/json" -H "Content-Type: application/json" -d '{"txt":"test"}'  It seems like it would be nicer to have a single URL that accepts both content-types and have the model parsed out appropriately. But the above code currently fails with either: {"detail":[{"loc":["body","txt"],"msg":"field required","type":"value_error.missing"}]}  or {"detail":"There was an error parsing the body"}  if I post to the "wrong" endpoint, e.g. form encoding posted to /json.  For bonus points; I'd also like to support form-data encoded parameters as it seems related (my txt can get rather long in practice), but might need to turn it into another question if it's sufficiently different.

Answer: FastAPI can't route based on Content Type, you'd have to check that in the request and parse appropriately: @app.post('/') async def route(req: Request) -> Response: if req.headers['Content-Type'] == 'application/json': item = MyItem(** await req.json()) elif req.headers['Content-Type'] == 'multipart/form-data': item = MyItem(** await req.form()) elif req.headers['Content-Type'] == 'application/x-www-form-urlencoded': item = MyItem(** await req.form()) return Response(content=item.json())  There seems to be an open issue on GitHub regarding this functionality
Question: I'm trying to calculate the eigenvectors and eigenvalues of this matrix  import numpy as np la = 0.02 mi = 0.08 n = 500 d1 = np.full(n, -(la+mi), np.double) d1[0] = -la d1[-1] = -mi d2 = np.full(n-1, la, np.double) d3 = np.full(n-1, mi, np.double) A = np.diagflat(d1) + np.diagflat(d2, -1) + np.diag(d3, 1) e_values, e_vectors = np.linalg.eig(A)  If I set the dimensions of the matrix to n < 110 the output is fine. However, if I set it to n >= 110 both the eigenvalues and the eigenvector components become complex numbers with significant imaginary parts. Why does this happen? Is it supposed to happen? It is very strange behavior and frankly I'm kind of stuck.

Answer: What you are seeing appears to be fairly normal roundoff error. This is an unfortunate result of storing floating point numbers with a finite precision. It naturally gets relatively worse for large problems. Here is a plot of the real vs. imaginary components of the eigenvalues:  You can see that the imaginary numbers are effectively noise. This is not to say that they are not important. Here is a plot of the imaginary vs. real part, showing that the ratio can get as large as 0.06 in the worst case:  This ratio changes with respect to the absolute and relative quantities la and mi. If you multiply both by 10, you get  If you keep la = 0.02 and set mi = 0.8, you get a smaller imaginary part:   Things get really weird when you do the opposite, and increase la by a factor of 10, keeping mi as-is:  The relative precision of the calculation decreases for smaller eigenvalues, so this is not too surprising. Given the relatively small magnitudes of the imaginary parts (at least for the important eigenvalues), you can either take the magnitude or the real part of the result since you know that all the eigenvalues are real.
Question: I have a data frame, which I created in pandas, grouping by date and summarizing by rides.   date rides 0 2019-01-01 247279 1 2019-01-02 585996 2 2019-01-03 660631 3 2019-01-04 662011 4 2019-01-05 440848 .. ... ... 451 2020-03-27 218499 452 2020-03-28 143305 453 2020-03-29 110833 454 2020-03-30 207743 455 2020-03-31 199623 [456 rows x 2 columns]  My date column is in datetime64[ns].  date datetime64[ns] rides int64 dtype: object  Now I would like to create another data frame, grouping by month and year (I have data form 2019 and 2020) and summarize by rides.  Ideal output: Year Month Rides 2019 January 2000000 2020 March 1000000 

Answer: you can groupby and get the dt.year and the dt.month_name from the column date. print (df.groupby([df['date'].dt.year.rename('year'), df['date'].dt.month_name().rename('month')]) ['rides'].sum().reset_index()) year month rides 0 2019 January 2596765 1 2020 March 880003
Question: Suppose I have two tensors: a = torch.randn(10, 1000, 1, 4) b = torch.randn(10, 1000, 6, 4)  Where the third index is the index of a vector. I want to take the dot product between each vector in b with respect to the vector in a. To illustrate, this is what I mean: dots = torch.Tensor(10, 1000, 6, 1) for b in range(10): for c in range(1000): for v in range(6): dots[b,c,v] = torch.dot(b[b,c,v], a[b,c,0])  How would I achieve this using torch functions?

Answer: a = torch.randn(10, 1000, 1, 4) b = torch.randn(10, 1000, 6, 4) c = torch.sum(a * b, dim=-1) print(c.shape)  torch.Size([10, 1000, 6]) c = c.unsqueeze(-1) print(c.shape)  torch.Size([10, 1000, 6, 1])
Question: For a current project, I am planning to filter a JSON file by timeranges by running several loops, each time with a slightly shifted range. The code below however yields the error TypeError: Invalid comparison between dtype=datetime64[ns] and date for line after_start_date = df["Date"] >= start_date. I have already tried to modify the formatting of the dates both within the Python code as well as the corresponding JSON file. Is there any smart tweak to align the date types/formats? The JSON file has the following format: [ {"No":"121","Stock Symbol":"A","Date":"05/11/2017","Text Main":"Sample text"} ]  And the corresponding code looks like this: import string import json import pandas as pd import datetime from dateutil.relativedelta import * # Loading and reading dataset file = open("Glassdoor_A.json", "r") data = json.load(file) df = pd.json_normalize(data) df['Date'] = pd.to_datetime(df['Date']) # Create an empty dictionary d = dict() # Filtering by date start_date = datetime.date.fromisoformat('2017-01-01') end_date = datetime.date.fromisoformat('2017-01-31') for i in df.iterrows(): start_date += relativedelta(months=+3) end_date += relativedelta(months=+3) print(start_date) print(end_date) after_start_date = df["Date"] >= start_date before_end_date = df["Date"] <= end_date between_two_dates = after_start_date & before_end_date filtered_dates = df.loc[between_two_dates] print(filtered_dates) 

Answer: You can use pd.to_datetime('2017-01-31') instead of datetime.date.fromisoformat('2017-01-31'). I hope this helps!
Question: I was learning Python in Codecademy and they were talking about using the open() function for CSV files. I couldn't really understand what the argument newline='' meant for the code. import csv with open('addresses.csv', newline='') as addresses_csv: address_reader = csv.DictReader(addresses_csv, delimiter=';') for row in address_reader: print(row['Address']) 

Answer: In your csv.DictReader function, you are iterating over lines in addresses.csv and mapping each row to a dict. Check the quoted fields in the csv file, and see if there are any escape sequences for ending a line '\r\n' - notice what happens when you include the newline parameter as shown in your code versus when you don't. Not including the newline parameter will probably add an extra line ending which you don't want. Including the newline parameter allows the csv module to handle the line endings itself - replicating the format as defined in your csv.
Question: Here is an simple example that gets min, max, and avg values from a list. The two functions below have same result. I want to know the difference between these two functions. And why use itertools.tee()? What advantage does it provide? from statistics import median from itertools import tee purchases = [1, 2, 3, 4, 5] def process_purchases(purchases): min_, max_, avg = tee(purchases, 3) return min(min_), max(max_), median(avg) def _process_purchases(purchases): return min(purchases), max(purchases), median(purchases) def main(): stats = process_purchases(purchases=purchases) print("Result:", stats) stats = _process_purchases(purchases=purchases) print("Result:", stats) if __name__ == '__main__': main() 

Answer: Iterators can only be iterated once in python. After that they are "exhausted" and don't return more values. You can see this in functions like map(), zip(), filter() and many others: purchases = [1, 2, 3, 4, 5] double = map(lambda n: n*2, purchases) print(list(double)) # [2, 4, 6, 8, 10] print(list(double)) # [] <-- can't use it twice  You can see the difference between your two functions if you pass them an iterator, such as the return value from map(). In this case _process_purchases() fails because min() exhausts the iterator and leaves no values for max() and median().  tee() takes an iterator and gives you two or more, allowing you to use the iterator passed into the function more than once: from itertools import tee from statistics import median purchases = [1, 2, 3, 4, 5] def process_purchases(purchases): min_, max_, avg = tee(purchases, 3) return min(min_), max(max_), median(avg) def _process_purchases(purchases): return min(purchases), max(purchases), median(purchases) double = map(lambda n: n*2, purchases) _process_purchases(double) # ValueError: max() arg is an empty sequence double = map(lambda n: n*2, purchases) process_purchases(double) # (2, 10, 6)
Question: I'm wondering how to get a message by its message id. I have tried discord.fetch_message(id) and discord.get_message(id), but both raise: Command raised an exception: AttributeError: module 'discord' has no attribute 'fetch_message'/'get_message' 

Answer: When getting a message, you're going to need an abc.Messageable object - essentially an object where you can send a message in, for example a text channel, a DM etc. Example: @bot.command() async def getmsg(ctx, msgID: int): # yes, you can do msg: discord.Message # but for the purposes of this, i'm using an int msg = await ctx.fetch_message(msgID) # you now have the message object from the id # ctx.fetch_message gets it from the channel # the command was executed in ################################################### @bot.command() async def getmsg(ctx, channel: discord.TextChannel, member: discord.Member): msg = discord.utils.get(await channel.history(limit=100).flatten(), author=member) # this gets the most recent message from a specified member in the past 100 messages # in a certain text channel - just an idea of how to use its versatility   References:  abc.Messageable TextChannel.history() TextChannel.fetch_message() Context.fetch_message()
Question: Linestring1 = LINESTRING (51.2176008 4.4177154, 51.21758 4.4178548, **51.2175729 4.4179023**, *51.21745162000732 4.41871738126533*) Linestring2 = LINESTRING (*51.21745162000732 4.41871738126533*, **51.2174025 4.4190475**, 51.217338 4.4194807, 51.2172511 4.4200562, 51.2172411 4.4201077, 51.2172246 4.4201654, 51.2172067 4.420205, 51.2171806 4.4202355, 51.2171074 4.4202929, 51.2170063 4.4203409, 51.2169564 4.4203641, 51.2168076 4.4204243, 51.2166588 4.4204833, 51.2159018 4.420431, 51.2154117 4.4203843)  Considering these two linestrings were cut from a bigger linestring, how to get the endpoint of a LineString? - Point(51.21745162000732 4.41871738126533) removed - The new last element of linestring 1 = “ 51.2175729 4.4179023 - The new first element of linestring 2 = “ 51.2174025 4.4190475 In short, I want to get the new last value of the first part (linestring1) and the new first value of the second part (linestring2), but without the point where I cut them. How can I make this work?

Answer: To get endpoints of a LineString, you just need to access its boundary property: from shapely.geometry import LineString line = LineString([(0, 0), (1, 1), (2, 2)]) endpoints = line.boundary print(endpoints) # MULTIPOINT (0 0, 2 2) first, last = line.boundary print(first, last) # POINT (0 0) POINT (2 2)  Alternatively, you can get the first and the last points from the coords cordinate sequence: from shapely.geometry import Point first = Point(line.coords[0]) last = Point(line.coords[-1]) print(first, last) # POINT (0 0) POINT (2 2)   In your specific case, though, as you want to remove the last point of the first line, and the first point of the second line, and only after that get the endpoints, you should construct new LineString objects first using the same coords property: from shapely.wkt import loads first_line = loads("LINESTRING (51.2176008 4.4177154, 51.21758 4.4178548, 51.2175729 4.4179023, 51.21745162000732 4.41871738126533)") second_line = loads("LINESTRING (51.21745162000732 4.41871738126533, 51.2174025 4.4190475, 51.217338 4.4194807, 51.2172511 4.4200562, 51.2172411 4.4201077, 51.2172246 4.4201654, 51.2172067 4.420205, 51.2171806 4.4202355, 51.2171074 4.4202929, 51.2170063 4.4203409, 51.2169564 4.4203641, 51.2168076 4.4204243, 51.2166588 4.4204833, 51.2159018 4.420431, 51.2154117 4.4203843)") first_line = LineString(first_line.coords[:-1]) second_line = LineString(second_line.coords[1:]) print(first_line.boundary[1], second_line.boundary[0]) # POINT (51.2175729 4.4179023) POINT (51.2174025 4.4190475)
Question: I keep getting this error when importing tensorflow as tf with the below error text:  ImportError: cannot import name 'export_saved_model' from 'tensorflow.python.keras.saving.saved_model'  Code used is simply: import tensorflow as tf  I have done:  uninstalled and installed tensorflow through pip and condo via anaconda cmd prompt Restarted and cleared outputs from kernel, closing the jupyternotebook and restarting my computer  Code used for tensorflow unstallation:  pip uninstall tensorflow or  conda uninstall -y tensorflow  Code used for tensorflow installation:  pip install tensorflow or  conda install tensorflow  I can't seem to figure why it keeps saying:   "ImportError: cannot import name 'export_saved_model' from 'tensorflow.python.keras.saving.saved_model'" 

Answer: Uninstalling and install again worked for me.  conda activate tf pip uninstall -y tensorflow-gpu pip install tensorflow-gpu  However, am still looking for the cause of this error. It was working just a few minutes ago but suddenly I have faced this error.
Question: I am trying to deploy my Django app which uses a machine learning model. And the machine learning model requires pytorch to execute. When i am trying to deploy it is giving me this error ERROR: Could not find a version that satisfies the requirement torch==1.5.0+cpu (from -r /tmp/build_4518392d43f43bc52f067241a9661c92/requirements.txt (line 23)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0) ERROR: No matching distribution found for torch==1.5.0+cpu (from -r /tmp/build_4518392d43f43bc52f067241a9661c92/requirements.txt (line 23)) ! Push rejected, failed to compile Python app. ! Push failed  My requirements.txt is  asgiref==3.2.7 certifi==2020.4.5.1 chardet==3.0.4 cycler==0.10.0 dj-database-url==0.5.0 Django==3.0.6 django-heroku==0.3.1 future==0.18.2 gunicorn==20.0.4 idna==2.9 imageio==2.8.0 kiwisolver==1.2.0 matplotlib==3.2.1 numpy==1.18.4 Pillow==7.1.2 psycopg2==2.8.5 pyparsing==2.4.7 python-dateutil==2.8.1 pytz==2020.1 requests==2.23.0 six==1.14.0 sqlparse==0.3.1 torch==1.5.0+cpu torchvision==0.6.0+cpu urllib3==1.25.9 whitenoise==5.0.1  And runtime.txt is python-3.7.5 However installing it on my computer is not giving any type of error when i use command pip install torch==1.5.0+cpu I am using python 3.7.5 and pip 20.0.2.  Complete code is here. How to solve this issue i really need to deploy my app. Thanks

Answer: PyTorch does not distribute the CPU only versions over PyPI. They are only available through their custom registry. If you select the CPU only version on PyTorch - Get Started Locally you get the following instructions: pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html  Since you're not manually executing the pip install, you cannot simply add the -f https://download.pytorch.org/whl/torch_stable.html. As an alternative, you can put it into your requirements.txt as a standalone line. It shouldn't really matter where exactly you put it, but it is commonly put at the very top. -f https://download.pytorch.org/whl/torch_stable.html asgiref==3.2.7 certifi==2020.4.5.1 chardet==3.0.4 cycler==0.10.0 dj-database-url==0.5.0 Django==3.0.6 django-heroku==0.3.1 future==0.18.2 gunicorn==20.0.4 idna==2.9 imageio==2.8.0 kiwisolver==1.2.0 matplotlib==3.2.1 numpy==1.18.4 Pillow==7.1.2 psycopg2==2.8.5 pyparsing==2.4.7 python-dateutil==2.8.1 pytz==2020.1 requests==2.23.0 six==1.14.0 sqlparse==0.3.1 torch==1.5.0+cpu torchvision==0.6.0+cpu urllib3==1.25.9 whitenoise==5.0.1
Question: What is the best way to read .tsv file with header in pyspark and store it in a spark data frame. I am trying to use "spark.read.options" and "spark.read.csv" commands however no luck. Thanks. Regards, Jit

Answer: Well you can directly read the tsv file without providing external schema if there is header available as: df = spark.read.csv(path, sep=r'\t', header=True).select('col1','col2')  Since spark is lazily evaluated it'll read only selected columns. Hope it helps.
Question: When I enter option + shift + o on my Mac in a Python file in VSCode, I am given two options - "Sort imports" and "Organize Imports". They both organize the inputs nicely but in a different way, so I can keep flipping back and forth between the two of them. Why are there two different commands for this and is one preferred (i.e. more PEP8 compliant) than the other?

Answer: There are two because the Python extension created the Sort Imports command before VS Code introduced the Organize Imports command. So the Sort Imports command is the more "official" one for now. There is an open issue, though, to transition over to Organize Imports at some point. Feel free to 👍 the issue if you would like to help bump its priority.