eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm having a hard time understanding how the AsyncIOScheduler works, and how is it non blocking? If my job is executing a blocking function, will the AsyncIOScheduler be blocking? And what if I use AsyncIOScheduler with ThreadPoolExecutor? How does that work? Can I await the job execution?

Answer: So, in APScheduler there are 3 important components:  The Scheduler The Executor(s) The Datastore(s)  For this question, only 1 and 2 are relevant. The Scheduler is simply who decides when to call the jobs based on their interval settings, in the case of AsyncIOScheduler it uses asyncio to make the waiting period non blocking. It runs in the same process and thread as the main. It is very useful if your application is already running on an asyncio loop since it saves the overhead of running a new proccess/thread. Now, when a job needs to be executed, is the Executor who is called, in the case of AsyncIOScheduler, by default is uses AsyncIOExecutor, which runs in the same thread and process as the Scheduler IF the job function is signed as async, else it uses asyncio's run_in_executor which runs it in a thread pool. Which brings us to the last question, what happens if we use AsyncIOScheduler with ThreadPoolExecutor? Well, technically is the same as using the default Executor with a non async function, it will run it in a thread pool, but the scheduler will remain in the main thread.
Question: The zipfile.ZipFile documentation says that ZIP_DEFLATED can be used as compression method only if zlib is available, but neither zipfile module specification nor zlib module specification says anything about when zlib might not be available, or how to check for its availability. I work on Windows and when I install any version of Python, zlib module is available. Is this different in Linux? Does zlib need to be installed separately? Also, what is the proper way to check for zlib availability? Is import zlib going to raise an ImportError if it is not available? In oher words, is this the correct way to use zipfile? try: import zlib except ImportError: zlib = None compression = zipfile.ZIP_STORED if zlib is None else zipfile.ZIP_DEFLATED with zipfile.ZipFile(file, mode, compression) as zf: ... 

Answer:  On Ubuntu if you install Python 3 using apt, e.g. sudo apt install python3.8, zlib will be installed as a dependency. Another way is to install Python 3 from source code. In this case, you need to install all prerequisites, including zlib1g-dev, (and this action is sometimes forgotten to do) and then compile and install python as sudo make install. Full instructions here Yes, if zlib is not available import zlib will raise exception, such as  >>> import zlib Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'zlib  There are examples of code like this in the Python standard library. e.g. in zipfile.py: try: import zlib # We may need its compression method crc32 = zlib.crc32 except ImportError: zlib = None crc32 = binascii.crc32
Question: I'm trying to solve some simple captcha using OpenCV and pytesseract. Some of captcha samples are:     I tried to the remove the noisy dots with some filters: import cv2 import numpy as np import pytesseract img = cv2.imread(image_path) _, img = cv2.threshold(img, 127, 255, cv2.THRESH_BINARY) img = cv2.morphologyEx(img, cv2.MORPH_OPEN, np.ones((4, 4), np.uint8), iterations=1) img = cv2.medianBlur(img, 3) img = cv2.medianBlur(img, 3) img = cv2.medianBlur(img, 3) img = cv2.medianBlur(img, 3) img = cv2.GaussianBlur(img, (5, 5), 0) cv2.imwrite('res.png', img) print(pytesseract.image_to_string('res.png'))  Resulting tranformed images are:     Unfortunately pytesseract just recognizes first captcha correctly. Any other better transformation? Final Update: As @Neil suggested, I tried to remove noise by detecting connected pixels. To find connected pixels, I found a function named connectedComponentsWithStats, whichs detect connected pixels and assigns group (component) a label. By finding connected components and removing the ones with small number of pixels, I managed to get better overall detection accuracy with pytesseract. And here are the new resulting images:    

Answer: I've taken a much more direct approach to filtering ink splotches from pdf documents. I won't share the whole thing it's a lot of code, but here is the general strategy I adopted:  Use Python Pillow library to get an image object where you can manipulate pixels directly. Binarize the image. Find all connected pixels and how many pixels are in each group of connected pixels. You can do this using the minesweeper algorithm. Which is easy to search for. Set some threshold value of pixels that all legitimate letters are expected to have. This will be dependent on your image resolution. replace all black pixels in groups below the threshold with white pixels. Convert back to image.
Question: I have a dataframe below df = pd.DataFrame({ 'ID': ['James', 'James', 'James', 'James', 'Max', 'Max', 'Max', 'Max', 'Max', 'Park', 'Park','Park', 'Park', 'Tom', 'Tom', 'Tom', 'Tom'], 'From_num': [578, 420, 420, 'Started', 298, 78, 36, 298, 'Started', 28, 28, 311, 'Started', 60, 520, 99, 'Started'], 'To_num': [96, 578, 578, 420, 36, 298, 78, 36, 298, 112, 112, 28, 311, 150, 60, 520, 99], 'Date': ['2020-05-12', '2020-02-02', '2020-02-01', '2019-06-18', '2019-08-26', '2019-06-20', '2019-01-30', '2018-10-23', '2018-08-29', '2020-05-21', '2020-05-20', '2019-11-22', '2019-04-12', '2019-10-16', '2019-08-26', '2018-12-11', '2018-10-09']})  and it is like this:  ID From_num To_num Date 0 James 578 96 2020-05-12 1 James 420 578 2020-02-02 2 James 420 578 2020-02-01 # Drop the this duplicated row (ignore date) 3 James Started 420 2019-06-18 4 Max 298 36 2019-08-26 5 Max 78 298 2019-06-20 6 Max 36 78 2019-01-30 7 Max 298 36 2018-10-23 8 Max Started 298 2018-08-29 9 Park 28 112 2020-05-21 10 Park 28 112 2020-05-20 # Drop this duplicate row (ignore date) 11 Park 311 28 2019-11-22 12 Park Started 311 2019-04-12 13 Tom 60 150 2019-10-16 14 Tom 520 60 2019-08-26 15 Tom 99 520 2018-12-11 16 Tom Started 99 2018-10-09  There are some consecutive duplicated values (ignore the Date value) within each 'ID'(Name), e.g. line 1 and 2 for James, the From_num are both 420, same as line 9 and 10, I wish to drop the 2nd duplicated row and keep the first. I wrote loop conditions, but it is very redundant and slow, I assume there might be easier way to do this, so please help if you have ideas. Great thanks. The expected result is like this:  ID From_num To_num Date 0 James 578 96 2020-05-12 1 James 420 578 2020-02-02 2 James Started 420 2019-06-18 3 Max 298 36 2019-08-26 4 Max 78 298 2019-06-20 5 Max 36 78 2019-01-30 6 Max 298 36 2018-10-23 7 Max Started 298 2018-08-29 8 Park 28 112 2020-05-21 9 Park 311 28 2019-11-22 10 Park Started 311 2019-04-12 11 Tom 60 150 2019-10-16 12 Tom 520 60 2019-08-26 13 Tom 99 520 2018-12-11 14 Tom Started 99 2018-10-09 

Answer: It's a bit late, but does this do what you wanted? This drops consecutive duplicates ignoring "Date". t = df[['ID', 'From_num', 'To_num']] df[(t.ne(t.shift())).any(axis=1)] ID From_num To_num Date 0 James 578 96 2020-05-12 1 James 420 578 2020-02-02 3 James Started 420 2019-06-18 4 Max 298 36 2019-08-26 5 Max 78 298 2019-06-20 6 Max 36 78 2019-01-30 7 Max 298 36 2018-10-23 8 Max Started 298 2018-08-29 9 Park 28 112 2020-05-21 11 Park 311 28 2019-11-22 12 Park Started 311 2019-04-12 13 Tom 60 150 2019-10-16 14 Tom 520 60 2019-08-26 15 Tom 99 520 2018-12-11 16 Tom Started 99 2018-10-09  This drops rows with index values 2 and 10.
Question: I am trying to make my discord bot react to its own message, pretty much. The system works like this: A person uses the command !!bug - And gets a message in DM', she/she is supposed to answer those questions. And then whatever he/she answered, it will be transferred an embedded message to an admin text-channel. But I need to add 3 emojis, or react with three different emojis. And depending on what the admin chooses, it will transfer the message once more. So if an admin reacts to an emoji that equals to "fixed", it will be moved to a "fixed" text-channel (the entire message). I have done a lot of research about this, but only found threads about the old discord.py, meaning await bot.add_react(emoji) - But as I have understood it, that no longer works! Here is my code: import discord from discord.ext import commands import asyncio TOKEN = '---' bot = commands.Bot(command_prefix='!!') reactions = [":white_check_mark:", ":stop_sign:", ":no_entry_sign:"] @bot.event async def on_ready(): print('Bot is ready.') @bot.command() async def bug(ctx, desc=None, rep=None): user = ctx.author await ctx.author.send('```Please explain the bug```') responseDesc = await bot.wait_for('message', check=lambda message: message.author == ctx.author, timeout=300) description = responseDesc.content await ctx.author.send('````Please provide pictures/videos of this bug```') responseRep = await bot.wait_for('message', check=lambda message: message.author == ctx.author, timeout=300) replicate = responseRep.content embed = discord.Embed(title='Bug Report', color=0x00ff00) embed.add_field(name='Description', value=description, inline=False) embed.add_field(name='Replicate', value=replicate, inline=True) embed.add_field(name='Reported By', value=user, inline=True) adminBug = bot.get_channel(733721953134837861) await adminBug.send(embed=embed) # Add 3 reaction (different emojis) here bot.run(TOKEN) 

Answer: In discord.py@rewrite, you have to use discord.Message.add_reaction: emojis = ['emoji 1', 'emoji_2', 'emoji 3'] adminBug = bot.get_channel(733721953134837861) message = await adminBug.send(embed=embed) for emoji in emojis: await message.add_reaction(emoji)  Then, to exploit reactions, you'll have to use the discord.on_reaction_add event. This event will be triggered when someone reacts to a message and will return a Reaction object and a User object: @bot.event async def on_reaction_add(reaction, user): embed = reaction.embeds[0] emoji = reaction.emoji if user.bot: return if emoji == "emoji 1": fixed_channel = bot.get_channel(channel_id) await fixed_channel.send(embed=embed) elif emoji == "emoji 2": #do stuff elif emoji == "emoji 3": #do stuff else: return   NB: You'll have to replace "emoji 1", "emoji 2" and "emoji 3" with your emojis. add_reactions accepts:  Global emojis (eg. 😀) that you can copy on emojipedia Raw unicode (as you tried) Discord emojis: \N{EMOJI NAME} Custom discord emojis (There might be some better ways, I've came up with this after a small amount a research) async def get_emoji(guild: discord.Guild, arg): return get(ctx.guild.emojis, name=arg)
Question: I am trying to download a csv file from an s3 bucket using the s3fs library. I have noticed that writing a new csv using pandas has altered data in some way. So I want to download the file directly in its raw state. The documentation has a download function but I do not understand how to use it: download(self, rpath, lpath[, recursive]): Alias of FilesystemSpec.get. Here's what I tried: import pandas as pd import datetime import os import s3fs import numpy as np #Creds for s3 fs = s3fs.S3FileSystem(key=mykey, secret=mysecretkey) bucket = "s3://mys3bucket/mys3bucket" files = fs.ls(bucket)[-3:] #download files: for file in files: with fs.open(file) as f: fs.download(f,"test.csv") AttributeError: 'S3File' object has no attribute 'rstrip' 

Answer: for file in files: fs.download(file,'test.csv')  Modified to download all files in the directory: import pandas as pd import datetime import os import s3fs import numpy as np #Creds for s3 fs = s3fs.S3FileSystem(key=mykey, secret=mysecretkey) bucket = "s3://mys3bucket/mys3bucket" #files references the entire bucket. files = fs.ls(bucket) for file in files: fs.download(file,'test.csv')
Question: In python, is there a way to retrieve the value of an env variable from a specific .env file? For example, I have multiple .env files as follows: .env.a .env.a ... And I have a variable in .env.b called INDEX=4. I tried receiving the value of INDEX by doing the following: import os os.getenv('INDEX')  But this value returns None. Any suggestions?

Answer: This is a job for ConfigParser or ConfigObj. ConfigParser is built into the Python standard library, but has the drawback that it REALLY wants section names. Below is for Python3. If you're still using Python2, then use import ConfigParser import configparser config = configparser.ConfigParser() config.read('env.b') index = config['mysection']['INDEX']  where env.b for ConfigParser is [mysection] INDEX=4  And using ConfigObj: import configobj config = configobj.ConfigObj('env.b') index = config['INDEX']  where env.b for ConfigObj is INDEX=4
Question: I created a virtual environment in vscode in a folder called server by typing: python -m venv env  And I opened the server folder, select interpreter Python 3.8.1 64-bit('env':venv) then I got following error:  I can't find any solution to this and I am stuck for hours.

Answer: It seems that it is going to activate the environment through a powershell script. And running such scripts is turned off by default. Also, usually a virtual environment is activated through cmd and .bat script. You could either turn on running powershell script or make VS Code activate an environment through cmd and .bat file. The first way - using cmd instead of Powershell I just checked it in my PC and VS Code doesn't use Powershell at all. It activate an environment with cmd instead of Powershell. Probably it is worth to check VS Code settings, set cmd as a default terminal. It is probably such an option in the main settings.json (you can open it through ctrl+shift+p and type 'open settings (JSON)'): "terminal.integrated.shell.windows": "C:\\Windows\\System32\\cmd.exe",. The second way - changing Powershell execution policy In order to change Powershell execution policy you can add "terminal.integrated.shellArgs.windows": ["-ExecutionPolicy", "Bypass"] to your main VS Code settings. Also you can open a Powershell window as administrator and type the following: Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser  Then respond y to any questions.
Question: I'm reading a table from csv and only want a subset of the columns. The list I'm using to subset contains field names that may not exist in the table I'm reading. For example: # contents of sample.csv: #a,b,c #1,2,3 #4,5,6 subset = ['a', 'c', 'd']  I'd like to return the following, using pandas.read_csv and the subset, but this raises an error: pd.read_csv(sample.csv, usecols=subset) a c 1 3 4 6 ValueError: Usecols do not match columns, columns expected but not found: ['d']  I think I might be able to do with a callable value for usecols, but am not sure how to implement.

Answer: Use a callable checking if the column is in the subset subset = ['a', 'c', 'd'] df = pd.read_csv('sample.csv', usecols=lambda x: x in subset) a c 0 1 3 1 4 6
Question: I have the following code which plots a bar chart (1 series), but I need the bars to be coloured blue if the 'Net' value is positive, and red if its negative: import pandas as pd import plotly.graph_objects as go df = pd.DataFrame({ 'Net':[15,20,-10,-15], 'Date':['07/14/2020','07/15/2020','07/16/2020','07/17/2020'] }) df['Date'] = pd.to_datetime(df['Date']) fig = go.Figure(data=[go.Bar(name='Net', x=df['Date'], y=df['Net'])]) fig.update_layout(barmode='stack') fig.show() 

Answer: You can check documentation here. Full code as following import pandas as pd import plotly.graph_objects as go import numpy as np # Data df = pd.DataFrame({ 'Net':[15,20,-10,-15], 'Date':['07/14/2020','07/15/2020','07/16/2020','07/17/2020'] }) df['Date'] = pd.to_datetime(df['Date']) ## here I'm adding a column with colors df["Color"] = np.where(df["Net"]<0, 'red', 'green') # Plot fig = go.Figure() fig.add_trace( go.Bar(name='Net', x=df['Date'], y=df['Net'], marker_color=df['Color'])) fig.update_layout(barmode='stack') fig.show()
Question: I have a pandas DataFrame that includes a column of text, and I would like to vectorize the text using scikit-learn's CountVectorizer. However, the text includes missing values, and so I would like to impute a constant value before vectorizing. My initial idea was to create a Pipeline of SimpleImputer and CountVectorizer: import pandas as pd import numpy as np df = pd.DataFrame({'text':['abc def', 'abc ghi', np.nan]}) from sklearn.impute import SimpleImputer imp = SimpleImputer(strategy='constant') from sklearn.feature_extraction.text import CountVectorizer vect = CountVectorizer() from sklearn.pipeline import make_pipeline pipe = make_pipeline(imp, vect) pipe.fit_transform(df[['text']]).toarray()  However, the fit_transform errors because SimpleImputer outputs a 2D array and CountVectorizer requires 1D input. Here's the error message: AttributeError: 'numpy.ndarray' object has no attribute 'lower'  QUESTION: How can I modify this Pipeline so that it will work? NOTE: I'm aware that I can impute missing values in pandas. However, I would like to accomplish all preprocessing in scikit-learn so that the same preprocessing can be applied to new data using Pipeline.

Answer: The best solution I have found is to insert a custom transformer into the Pipeline that reshapes the output of SimpleImputer from 2D to 1D before it is passed to CountVectorizer. Here's the complete code: import pandas as pd import numpy as np df = pd.DataFrame({'text':['abc def', 'abc ghi', np.nan]}) from sklearn.impute import SimpleImputer imp = SimpleImputer(strategy='constant') from sklearn.feature_extraction.text import CountVectorizer vect = CountVectorizer() # CREATE TRANSFORMER from sklearn.preprocessing import FunctionTransformer one_dim = FunctionTransformer(np.reshape, kw_args={'newshape':-1}) # INCLUDE TRANSFORMER IN PIPELINE from sklearn.pipeline import make_pipeline pipe = make_pipeline(imp, one_dim, vect) pipe.fit_transform(df[['text']]).toarray()  It has been proposed on GitHub that CountVectorizer should allow 2D input as long as the second dimension is 1 (meaning: a single column of data). That modification to CountVectorizer would be a great solution to this problem!
Question: As you can read on the title, I'm currently trying to make the code section collapsing without collapsing also the results section. For example without the collapse the title/code/result sections look like this:  If I try to "collapse" the section will look like this:  I'm looking for a solution where I can make collapse the code but not the results sections

Answer: You can start the cell with #@title  Then, you can double click the title. It will hide the code part, but keep the output still visible. For example #@title My title greet = "Hello" print(greet)  Double clicking "My title" will still show the output "Hello".
Question: I've made a classproperty descriptor and whenever I use a function decorated with it, I get multiple pylint inspection errors. Here is a sample class with a sample decorated function: class Bar: """ Bar documentation. """ # pylint: disable=no-method-argument @classproperty def foo(): """ Retrieve foo. """ return "foo"  Thanks to the descriptor, I can call Bar.foo and get the string foo returned. Unfortunately, whenever I use functions like this with slightly more complex items (e.g. functions which return instances of objects), pylint starts complaining about things such as no-member or unexpected-keyword-arg, simply because it thinks Bar.foo is a method, rather than a wrapped classproperty object. I would like to disable warnings for any code that uses my function - I definitely can't allow having to write # pylint: disable every single time I use the classproperty-wrapped methods. How can I do it with pylint? Or maybe I should switch to use a different linter instead? Here is an example of a warning generated because of the reasons above: class Bar: """ Bar documentation. """ # pylint: disable=no-method-argument @classproperty def foo(): """ Retrieve an object. """ return NotImplementedError("Argument") print(Bar.foo.args)  pylint complains that E1101: Method 'foo' has no 'args' member (no-member) (even though I know it definitely has), and I would like to completely disable some warnings for any module/class/function that uses Bar.foo.args or similar. For anyone interested, here is a minimal implementation of a classproperty descriptor: class classproperty: """ Minimal descriptor. """ # pylint: disable=invalid-name def __init__(self, func): self._func = func def __get__(self, _obj, _type): return self._func() 

Answer: I have managed to create a dirty hack by type-hinting the items as None: class Bar: """ Bar documentation. """ # pylint: disable=no-method-argument,function-redefined,too-few-public-methods foo: None @classproperty def foo(): """ Retrieve an object. """ return NotImplementedError("Argument")  I would rather avoid having code like this because I can't actually import the items which should be type-hinted due to the circular imports issue (hence None), but it tricks pylint well.
Question: I have the following goal: I have a python script, which should be running in my custom Anaconda environment. And this process needs to be automatizated. The first thing I've tried was to create an .exe file of my script using pyinstaller in the Anaconda command prompt, opened in my environment. And put the .exe into Windows Task Scheduler. But I did not succeeded cause my script seems to be too complex, contain too many imports so pyinstaller didn't create the .exe. The next thing I thought of was an attempt to run my script using Windows CMD with appropriate attributes, and also put it into Windows Task Scheduler. Now my question is if there is a way to set up Task Scheduler so it could run CMD with attributes, which would activate my environment and with this environment run my script right away? I need this to be done automatically once a day at a given time. Update 3: am I blind or what?  I mean, here it is:

Answer: You could  Create a .bat file (e.g. run_python_script.bat) with contents shown below. Create task in "Task Scheduler" to run the .bat file.  1.a. The .bat file contents with conda environments  Check your <condapath>. Your conda.exe is located at <condapath>/Scripts. Put into your .bat file  call "<condapath>\Scripts\activate.bat" <env_name> & cd "<folder_for_your_py_script>" & python <scriptname.py> [<arguments>]   <env_name> is the name of the conda environment. <folder_for_your_py_script> is the folder that contains <scriptname.py> <scriptname.py> is the script you want to start. [<arguments>] represent the optional arguments (if you need to give arguments to your script)  1.b. The .bat file contents with venv "<path_to_python_exe>" "<path_to_python_script>" [<arguments>]  where  <path_to_python_exe> is the path to your python executable. If you are using a virtual environment (venv), then use the python.exe found in the /venv/Scripts folder <path_to_python_script> is the path to your python script. [<arguments>] represent the optional arguments (if you need to give arguments to your script)  2. Creating task in Task Scheduler  Go to "Task Scheduler" -> "Create Basic Task" Give the name & timing info Add to the "Program/Script" the path to your run_python_script.bat.   Appendix: Creating venv with Anaconda It seems that conda create command does not create similar virtual environments as python -m venv command. To create normal python virtual environment with the venv  Check your <condapath>. Your conda.exe is located at <condapath>/Scripts. Create virtual environment to folder you want (let's call it venv_folder), by running following command in <venv_folder>  <condapath>\python.exe -m venv venv   Now, your <path_to_python_exe> will be <venv_folder>\venv\Scripts.python.exe. If you need to install packages to this virtual environment, you use  <venv_folder>\venv\Scripts.python.exe -m pip install <package_name>