eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Consider df: In [2098]: df = pd.DataFrame({'a': [1,2], 'b':[3,4]}) In [2099]: df Out[2099]: a b 0 1 3 1 2 4  Now, I try to append a list of values to df: In [2102]: df.loc[2] = [3, 4] In [2103]: df Out[2103]: a b 0 1 3 1 2 4 2 3 4  All's good so far. But now when I try to append a row with list of boolean values, it converts it into int: In [2104]: df.loc[3] = [True, False] In [2105]: df Out[2105]: a b 0 1 3 1 2 4 2 3 4 3 1 0  I know I can convert my df into str and can then append boolean values, like: In [2131]: df = df.astype(str) In [2133]: df.loc[3] = [True, False] In [2134]: df Out[2134]: a b 0 1 3 1 2 4 3 True False  But, I want to know the reason behind this behaviour. Why is it not automatically changing the dtypes of columns to object when I append boolean to it? My Pandas version is: In [2150]: pd.__version__ Out[2150]: '1.1.0' 

Answer: Why is it not automatically changing the dtypes of columns to object when I append boolean to it? Because the type are being upcasted (see upcasting), from the documentation:  Types can potentially be upcasted when combined with other types, meaning they are promoted from the current type (e.g. int to float).  Upcasting works according to numpy rules:  Upcasting is always according to the numpy rules. If two different dtypes are involved in an operation, then the more general one will be used as the result of the operation.  To understand how the numpy rules are applied you can use the function find_common_type, as below: res = np.find_common_type([bool, np.bool], [np.int32, np.int64]) print(res)  Output int64
Question: I want to just read and display an MP4 video using OpenCV, I wrote the following basic code for it: import cv2 input_video_path = './Input Video/Input_video1.mp4' cap = cv2.VideoCapture(input_video_path) while(cap.isOpened()): ret, frame = cap.read() print(frame, ret) cv2.imshow("frame", frame) cap.release() cv2.destroyAllWindows()  When I run it, it reads 1st few frames and then all other frames are None: [[[ 7 14 27] [ 7 14 27] [ 7 14 27] ... ... [ 60 57 64] [ 70 62 64] [ 72 64 66]]] True None False Traceback (most recent call last): File "D:/Project/ML IP and Coding/Cynapto_Task/exploring_face_detection_methods.py", line 10, in <module> cv2.imshow("frame", frame) cv2.error: OpenCV(3.4.2) C:\Miniconda3\conda-bld\opencv-suite_1534379934306\work\modules\highgui\src\window.cpp:356: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'cv::imshow'  I also printed the ret variable which confirms the same behavior. I tried with 2-3 different videos and even with .avi format but got the same result. I searched online for a solution but could only find about codecs and installing codecs in my virtual environment. I don't have much knowledge about video file formats and codecs. Can someone help me with this? I am using: Python 3.7, OS: Windows, environment: conda, OpenCV v4.5.0

Answer: If you recieved ret as False it means that video reach end frame. If video isn't finished but you recieved False, it probably broken. Try this code: import cv2 input_video_path = './Input Video/Input_video1.mp4' cap = cv2.VideoCapture(input_video_path) while(cap.isOpened()): ret, frame = cap.read() print(frame, ret) if ret: cv2.imshow("frame", frame) cv2.waitKey(1) else: break cap.release() cv2.destroyAllWindows()
Question: I'm new to julia and I'm working on to rewrite julia code to python code. And I saw the some codes using .== expression. I couldn't understand what this means. So I searched it on web but couldn't find an answer. Can someone tell me what is .== in julia and its equivalent in python? fyi, it was written like below. x = sum(y .== 0) # y is array 

Answer: That's a Vectorized dot operation and is used to apply the operator to an array. You can do this for one dimensional lists in python via list comprehensions, but here it seems like you are just counting all zeroes, so >>> y = [0,1,1,1,0] >>> sum(not bool(v) for v in y) 2  Other packages like numpy or pandas will vectorize operators, so something like this will do >>> import numpy as np >>> y = np.array([0,1,1,1,0]) >>> (y == 0).sum() 2 >>> >>> import pandas as pd >>> df=pd.DataFrame([[0,1,2,3], [1,2,3,0], [2,3,4,0]]) >>> (df==0).sum() 0 1 1 0 2 0 3 2 dtype: int64 >>> (df==0).sum().sum() 3
Question: I have this code in Python: import discord client = commands.Bot(command_prefix='!') @client.event async def on_voice_state_update(member): channel = client.get_channel(channels_id_where_i_want_to_send_message)) response = f'Hello {member}!' await channel.send(response) client.run('bots_token')  And I want the bot to delete its own message. For example, after one minute, how do I do it?

Answer: There is a better way than what Dean Ambros and Dom suggested, you can simply add the key-word argument delete_after in .send await ctx.send('whatever', delete_after=60.0)  reference
Question: Is there a way to load a pytorch DataLoader (torch.utils.data.Dataloader) entirely into my GPU? Now, I load every batch separately into my GPU. CTX = torch.device('cuda') train_loader = torch.utils.data.DataLoader( train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=0, ) net = Net().to(CTX) criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(net.parameters(), lr=LEARNING_RATE) for epoch in range(EPOCHS): for inputs, labels in test_loader: inputs = inputs.to(CTX) # this is where the data is loaded into GPU labels = labels.to(CTX) optimizer.zero_grad() outputs = net(inputs) loss = criterion(outputs, labels) loss.backward() optimizer.step() print(f'training accuracy: {net.validate(train_loader, device=CTX)}/{len(train_dataset)}') print(f'validation accuracy: {net.validate(test_loader, device=CTX)}/{len(test_dataset)}')  where the Net.validate() function is given by def validate(self, val_loader, device=torch.device('cpu')): correct = 0 for inputs, labels in val_loader: inputs = inputs.to(device) labels = labels.to(device) outputs = torch.argmax(self(inputs), dim=1) correct += int(torch.sum(outputs==labels)) return correct   I would like to improve the speed by loading the entire dataset trainloader into my GPU, instead of loading every batch separately. So, I would like to do something like train_loader.to(CTX)  Is there an equivalent function for this? Because torch.utils.data.DataLoader does not have this attribute .to(). I work with an NVIDIA GeForce RTX 2060 with CUDA Toolkit 10.2 installed.

Answer: you can put your data of dataset in advance train_dataset.train_data.to(CTX) #train_dataset.train_data is a Tensor(input data) train_dataset.train_labels.to(CTX)  for example of minst import torch from torch.utils.data import DataLoader from torchvision import datasets from torchvision import transforms batch_size = 64 transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)) ]) train_data = datasets.MNIST( root='./dataset/minst/', train=True, download=False, transform=transform ) train_loader = DataLoader( dataset=train_data, shuffle=True, batch_size=batch_size ) train_data.train_data = train_data.train_data.to(torch.device("cuda:0")) # put data into GPU entirely train_data.train_labels = train_data.train_labels.to(torch.device("cuda:0")) # edit note for newer versions: use train_data.data and train_data.targets instead  I got this solution by using debugger...
Question: I have a variable which is grabbing a date object from a file. My aim is to add a timezone to this object so that it automatically changes the time based on the date its it then. So I expected it to add +1hour to it for dates in summertime (between march and october) and add +0hour in wintertime (between october and march).  dt_object = '20200901-01u30m30s' dt_object = datetime.datetime.strptime(dt_object, '%Y%m%d-%Hu%Mm%Ss') >>>print(dt_object) >>> 2020-09-01 01:30:30 timezone= 'Europe/Amsterdam' dt_object_tz = pytz.utc.localize(dt_object).astimezone(pytz.timezone(timezone)) timeDiff = dt_object_tz.utcoffset().total_seconds() official_time = pytz.utc.localize(dt_object_tz+datetime.timedelta(seconds=timeDiff)) >>>print(official_time) >>> 2020-09-01 03:30:30+00:00  As you can see this is a datetime object of september (so summertime!), I literally have no clue why it adds +2hours instead of 1 hour.... Can someone explain it and tell me what went wrong? I just want my datetime object to be timezone-aware so it autmatically changes from summer to wintertime based on the date in grabs.

Answer: Regarding pytz, note that there is zoneinfo in the standard lib. No need for a third party library for time zone handling with Python >= 3.9. Example usage.  Then, if your input represents wall time in some time zone, you can just localize. If the input represents UTC, you can set the tzinfo to UTC a bit more easily and then convert to local time using astimezone: from datetime import datetime, timezone import pytz s = '20200901-01u30m30s' local_tz = 'Europe/Amsterdam' # if s represents local time, just localize: dtobj_tz = pytz.timezone(local_tz).localize(datetime.strptime(s, '%Y%m%d-%Hu%Mm%Ss')) # datetime.datetime(2020, 9, 1, 1, 30, 30, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>) # if s represents UTC, set it directly: dtobj_utc = datetime.strptime(s, '%Y%m%d-%Hu%Mm%Ss').replace(tzinfo=timezone.utc) # ...and convert to desired tz: dtobj_tz = dtobj_utc.astimezone(pytz.timezone(local_tz)) # datetime.datetime(2020, 9, 1, 3, 30, 30, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>)
Question: from selenium import webdriver; browser= webdriver.Firefox(); browser.get('http://www.seleniumhq.org');  When I try to run this code, it gives me an error message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line.  Any thoughts-highly appreciated!

Answer: This error message... Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided, and no binary flag set on the command line.  ...implies that the GeckoDriver was unable to find the Firefox binary at the default location. Additionally you haven't passed the moz:firefoxOptions.binary capability.  Solution Possibly within your system firefox is installed in a custom location and these cases you need to pass the absolute path of the Firefox binary through the moz:firefoxOptions.binary capability as follows: from selenium import webdriver from selenium.webdriver.firefox.options import Options options = Options() options.binary_location = r'C:\Program Files\Mozilla Firefox\firefox.exe' driver = webdriver.Firefox(executable_path=r'C:\WebDrivers\geckodriver.exe', options=options) driver.get('http://google.com/')   References You can find a couple of relevant detailed discussion in:  SessionNotCreatedException: Message: Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' InvalidArgumentException: Message: binary is not a Firefox executable error using GeckoDriver Firefox Selenium and Python Expected browser binary location, but unable to find binary in default location, no 'moz:firefoxOptions.binary' capability provided
Question: Consider:  st recent call last):  File "randomgg.py", line 1271, in \u003cmodule\u003e  client.run(token)  File "/usr/local/lib/python3.8/site-packages/discord/client.py", line 708, in run  return future.result()  File "/usr/local/lib/python3.8/site-packages/discord/client.py", line 687, in runner  await self.start(*args, **kwargs)  File "/usr/local/lib/python3.8/site-packages/discord/client.py", line 651, in start  await self.connect(reconnect=reconnect)  File "/usr/local/lib/python3.8/site-packages/discord/client.py", line 586, in connect  raise PrivilegedIntentsRequired(exc.shard_id) from None  discord.errors.PrivilegedIntentsRequired: Shard ID None is requesting privileged intents that have not been explicitly enabled in the developer portal. It is recommended to go to https://discord.com/developers/applications/ and explicitly enable the privileged intents within your application's page. If this is not possible, then consider disabling the privileged intents instead. \  My code was import aiohttp import discord import asyncio from collections import Counter import typing from discord.ext import commands import os from discord.ext.commands import has_permissions import random import json from discord import Status from asyncio import gather from discord.utils import get import datetime from discord.utils import get intents = discord.Intents.default() intents.members = True client = commands.Bot(command_prefix='.', intents=intents) client.remove_command('help') def check_if_it_is_me(ctx): return ctx.message.author.id == 465946367622381578 @client.event async def status_task(): while True: await client.change_presence(status=discord.Status.idle, activity=discord.Game('status1')) await asyncio.sleep(4) await client.change_presence(status=discord.Status.idle, activity=discord.Game('status2')) await asyncio.sleep(4) await client.change_presence(status=discord.Status.idle, activity=discord.Game('status3')) await asyncio.sleep(4) @client.event async def on_ready(): print(f'{client.user.name} is ready') client.loop.create_task(status_task()) @client.event async def on_member_join(member): mem_join = member.joined_at guild_create = member.created_at join_days = (mem_join - guild_create).days role = discord.utils.get(member.guild.roles, id=714805001918349344) channel = discord.utils.get(member.guild.channels, id=771081754038501376) if join_days < 10: await channel.send(f'{role.mention} {member} is suspicious of being an alt, he joined {join_days} after creating his account. Pls keep an eye on him') @client.event async def on_member_remove(member): pass  How can I fix this?

Answer: The error tells you exactly what to do.  Go to https://discord.com/developers/applications Navigate to your application Go to the Bot section Scroll down and enable SERVER MEMBERS INTENT
Question: My question here may seem really naive but I never found any clue about it on web resources. The question is, concerning install_requires argument for setup() function or setup.cfg file, is it a good practice to mention every package used, even python built-in ones such as os for example ? One can assume that any python environment has those common packages, so is it problematic to explicitly mention them in the setup, making it potentially over-verbose ? Thanks

Answer: install_requires should include non-standard library requirements, and constraints on their versions (as needed). For example, this would declare minimal versions for numpy and scipy, but allow any version of scikit-learn: setup( # ... install_requires=["numpy>=1.13.3", "scipy>=0.19.1", "scikit-learn"] )  Packages such as os, sys are part of Python's standard library, so should not be included.  As @sinoroc mentioned, only direct 3rd party dependencies should be declared here. Dependencies-of-your-dependencies are handled automatically. (For example, scikit-learn depends on joblib; when the former is required, the latter will be installed). A list of standard library packages are listed here: https://docs.python.org/3/library/   I've found it helpful to read other packages and see how their setup.py files are defined.  imbalanced-learn pandas
Question: I have a tensor t = torch.zeros((4, 5, 6)) How to check if it is on gpu or not, and send it to gpu and back?

Answer: From the pytorch forum use t.is_cuda, t.cuda(), t.cpu() t = torch.randn(2,2) t.is_cuda # returns False t = torch.randn(2,2).cuda() t.is_cuda # returns True t = t.cpu() t.is_cuda # returns False  When passing to and from gpu and cpu, new arrays are allocated on the relevant device.
Question: In playwright-python I know I can get an elementHandle using querySelector(). Example (sync): from playwright import sync_playwright with sync_playwright() as p: for browser_type in [p.chromium, p.firefox, p.webkit]: browser = browser_type.launch() page = browser.newPage() page.goto('https://duckduckgo.com/') element = page.querySelector('input[id=\"search_form_input_homepage\"]')  How do I get the an element relative to this based on this elementHandle? I.e. the parent, grandparent, siblings, children handles?

Answer: Original answer: Using querySelector() / querySelectorAll with XPath (XML Path Language) lets you retrieve the elementHandle (respectively a collection of handles). Generally speaking, XPath can be used to navigate through elements and attributes in an XML document. from playwright import sync_playwright with sync_playwright() as p: for browser_type in [p.chromium, p.firefox, p.webkit]: browser = browser_type.launch(headless=False) page = browser.newPage() page.goto('https://duckduckgo.com/') element = page.querySelector('input[id=\"search_form_input_homepage\"]') parent = element.querySelector('xpath=..') grandparent = element.querySelector('xpath=../..') siblings = element.querySelectorAll('xpath=following-sibling::*') children = element.querySelectorAll('xpath=child::*') browser.close()  Update (2022-07-22): It seems that browser.newPage() is deprecated, so in newer versions of playwright, the function is called browser.new_page() (note the different function name). Optionally create a browser context first (and close it afterwards) and call new_page() on that context. The way the children/parent/grandparent/siblings are accessed stays the same. from playwright import sync_playwright with sync_playwright() as p: for browser_type in [p.chromium, p.firefox, p.webkit]: browser = browser_type.launch(headless=False) context = browser.new_context() page = context.new_page() page.goto('https://duckduckgo.com/') element = page.querySelector('input[id=\"search_form_input_homepage\"]') parent = element.querySelector('xpath=..') grandparent = element.querySelector('xpath=../..') siblings = element.querySelectorAll('xpath=following-sibling::*') children = element.querySelectorAll('xpath=child::*') context.close() browser.close()
Question: Let df_1 and df_2 be: In [1]: import pandas as pd ...: df_1 = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}) ...: df_2 = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}) In [2]: df_1 Out[2]: a b 0 1 4 1 2 5 2 3 6  We add a row r to df_1: In [3]: r = pd.DataFrame({'a': ['x'], 'b': ['y']}) ...: df_1 = df_1.append(r, ignore_index=True) In [4]: df_1 Out[4]: a b 0 1 4 1 2 5 2 3 6 3 x y  We now remove the added row from df_1 and get the original df_1 back again: In [5]: df_1 = pd.concat([df_1, r]).drop_duplicates(keep=False) In [6]: df_1 Out[6]: a b 0 1 4 1 2 5 2 3 6 In [7]: df_2 Out[7]: a b 0 1 4 1 2 5 2 3 6  While df_1 and df_2 are identical, equals() returns False. In [8]: df_1.equals(df_2) Out[8]: False  Did reseach on SO but could not find a related question. Am I doing somthing wrong? How to get the correct result in this case? (df_1==df_2).all().all() returns True but not suitable for the case where df_1 and df_2 have different length.

Answer: Use pandas.testing.assert_frame_equal(df_1, df_2, check_dtype=True), which will also check if the dtypes are the same. (It will pick up in this case that your dtypes changed from int to 'object' (string) when you appended, then deleted, a string row; pandas did not automatically coerce the dtype back down to less expansive dtype.) AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="a") are different Attribute "dtype" are different [left]: object [right]: int64
Question: I use this code to test KFold and StratifiedKFold. import numpy as np from sklearn.model_selection import KFold,StratifiedKFold X = np.array([ [1,2,3,4], [11,12,13,14], [21,22,23,24], [31,32,33,34], [41,42,43,44], [51,52,53,54], [61,62,63,64], [71,72,73,74] ]) y = np.array([0,0,0,0,1,1,1,1]) sfolder = StratifiedKFold(n_splits=4,random_state=0,shuffle=False) floder = KFold(n_splits=4,random_state=0,shuffle=False) for train, test in sfolder.split(X,y): print('Train: %s | test: %s' % (train, test)) print("StratifiedKFold done") for train, test in floder.split(X,y): print('Train: %s | test: %s' % (train, test)) print("KFold done")  I found that StratifiedKFold can keep the proportion of labels, but KFold can't. Train: [1 2 3 5 6 7] | test: [0 4] Train: [0 2 3 4 6 7] | test: [1 5] Train: [0 1 3 4 5 7] | test: [2 6] Train: [0 1 2 4 5 6] | test: [3 7] StratifiedKFold done Train: [2 3 4 5 6 7] | test: [0 1] Train: [0 1 4 5 6 7] | test: [2 3] Train: [0 1 2 3 6 7] | test: [4 5] Train: [0 1 2 3 4 5] | test: [6 7] KFold done  It seems that StratifiedKFold is better, so should KFold not be used? When to use KFold instead of StratifiedKFold?

Answer: I think you should ask "When to use StratifiedKFold instead of KFold?". You need to know what "KFold" and "Stratified" are first.  KFold is a cross-validator that divides the dataset into k folds.   Stratified is to ensure that each fold of dataset has the same proportion of observations with a given label.  So, it means that StratifiedKFold is the improved version of KFold Therefore, the answer to this question is we should prefer StratifiedKFold over KFold when dealing with classification tasks with imbalanced class distributions.  FOR EXAMPLE Suppose that there is a dataset with 16 data points and imbalanced class distribution. In the dataset, 12 of data points belong to class A and the rest (i.e. 4) belong to class B. The ratio of class B to class A is 1/3. If we use StratifiedKFold and set k = 4, then, in each iteration, the training sets will include 9 data points from class A and 3 data points from class B while the test sets include 3 data points from class A and 1 data point from class B. As we can see, the class distribution of the dataset is preserved in the splits by StratifiedKFold while KFold does not take this into consideration.
Question: Thank you in advance for your help. My python code reads json input file and loads the data into a data frame, masks or changes on the data frame column specified by configuration and in the last stage, creates json output file. read json into data frame --> mask/change the df column ---> generate json  Input json: [ { "BinLogFilename": "mysql.log", "Type": "UPDATE", "Table": "users", "ServerId": 1, "BinLogPosition": 2111 }, { { "BinLogFilename": "mysql.log", "Type": "UPDATE", "Table": "users", "ServerId": null, "BinLogPosition": 2111 }, ... ]  when I load the above json into data frame, the data frame column "ServerId" has float values because it has null in few blocks of json input. The main central logic converts/fakes "ServerId" into another number, however the output contains float numbers. Output json: [ { "BinLogFilename": "mysql.log", "Type": "UPDATE", "Table": "users", "ServerId": 5627.0, "BinLogPosition": 2111 }, { "BinLogFilename": "mysql.log", "Type": "UPDATE", "Table": "users", "ServerId": null, "BinLogPosition": 2111 }, .... ]  masking logic df['ServerId'] = [fake.pyint() if not(pd.isna(df['ServerId'][index])) else np.nan for index in range(len(df['ServerId']))]  The challenge is, the output "ServerId" should contain only integers but unfortunately it contains floats. df['ServerId'] 0 9590.0 1 NaN 2 1779.0 3 1303.0 4 NaN  I found a answer to this problem, to use 'Int64' df['ServerId'] = df['ServerId'].astype('Int64') 0 8920 1 <NA> 2 9148 3 2434 4 <NA>  However using 'Int64', it converts NaN to NA and while writing back to json, i get an error as,  TypeError: Object of type NAType is not JSON serializable  with gzip.open(outputFile, 'w') as outfile: outfile.write(json.dumps(json_objects_list).encode('utf-8'))  Is it possible to keep NaN after converting to 'Int64' data type? If this is not possible, how can i fix the error?

Answer: Indeed, Pandas NA and NaT are not JSON serialisable by the built-in Python json library. But the Pandas DataFrame to_json() method will handle those values for you and convert them to JSON null. from pandas import DataFrame, Series, NA, NaT df = DataFrame({"ServerId" : Series([8920, NA, 9148, 2434, NA], dtype="Int64") }) s = df.to_json() # -> {"ServerId":{"0":8920,"1":null,"2":9148,"3":2434,"4":null}}