eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My issue is that when I execute poetry install, poetry update or poetry lock the process keeps running indefinitely. I tried using the -vvv flag to get output of what's happening and it looks like it gets stuck forever in the first install. My connection is good and all packages that I tried installing exist. I use version 1.2.1 but I cannot upgrade to newer versions because the format of the .lock file is different and our pipeline fails.

Answer: I found a clue in an issue on the GitHub repo. If you are using Linux you must delete all .lock files in the .cache/pypoetry dir in your user home directory. find ~/.cache/pypoetry -name '*.lock' -type f -delete  If the directory does not exist maybe is in another location. Then I recommend removing the generated .lock file in the project you were doing the installation.
Question: I have fine-tuned the T5-base model (from hugging face) on a new task where each input and target are sentences of 256 words. The loss is converging to low values however when I use the generate method the output is always too short. I tried giving minimal and maximal length values to the method but it doesn't seem to be enough. I suspect the issue is related to the fact that the sentence length before tokenization is 256 and after tokenization, it is not constant (padding is used during training to ensure all inputs are of the same size). Here is my generate method: model = transformers.T5ForConditionalGeneration.from_pretrained('t5-base') tokenizer = T5Tokenizer.from_pretrained('t5-base') generated_ids = model.generate( input_ids=ids, attention_mask=attn_mask, max_length=1024, min_length=256, num_beams=2, early_stopping=False, repetition_penalty=10.0 ) preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids][0] preds = preds.replace("<pad>", "").replace("
", "").strip().replace(" ", " ") target = [tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True) for t in reference][0] target = target.replace("<pad>", "").replace("
", "").strip().replace(" ", " ")  The inputs are created using tokens = tokenizer([f"task: {text}"], return_tensors="pt", max_length=1024, padding='max_length') inputs_ids = tokens.input_ids.squeeze().to(dtype=torch.long) attention_mask = tokens.attention_mask.squeeze().to(dtype=torch.long) labels = self.tokenizer([target_text], return_tensors="pt", max_length=1024, padding='max_length') label_ids = labels.input_ids.squeeze().to(dtype=torch.long) label_attention = labels.attention_mask.squeeze().to(dtype=torch.long) 

Answer: For whom it may concern, I found out the issue was with the max_length argument of the generation method. It limits the maximal number of tokens including the input tokens. In my case it was required to set max_new_tokens=1024 instead of the argument provided in the question.
Question: I have been using camelot for our project, but since 2 days I got following errorMessage. When trying to run following code snippet: import camelot tables = camelot.read_pdf('C:\\Users\\user\\Downloads\\foo.pdf', pages='1')  I get this error: DeprecationError: PdfFileReader is deprecated and was removed in PyPDF2 3.0.0. Use PdfReader instead.  I checked this file and it does use pdfFileReader: c:\ProgramData\Anaconda3\lib\site-packages\camelot\handlers.py I thought that I can specify the version of PyPDF2, but it will be installed automatically(because the library is used by camelot) when I install camelot. Do you think there is any solution to specify the version of PyPDF2 manually?

Answer: This is issues #339. While there will hopefully be soon a release including the fix, you can still do this: pip install 'PyPDF2<3.0'  after you've installed camelot. See https://github.com/camelot-dev/camelot/issues/339#issuecomment-1367331630 for details and screenshots.
Question: AttributeError: module 'cv2.aruco' has no attribute 'Dictionary_get' even after installing  opencv-python opencv-contrib-python  import numpy as np import cv2, PIL from cv2 import aruco import matplotlib.pyplot as plt import matplotlib as mpl import pandas as pd vid = cv2.VideoCapture(0) while (True): ret, frame = vid.read() #cv2.imshow('frame', frame) if cv2.waitKey(1) & 0xFF == ord('q'): break gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY) aruco_dict = aruco.Dictionary_get(aruco.DICT_6X6_250) parameters = aruco.DetectorParameters() corners, ids, rejectedImgPoints = aruco.detectMarkers(gray, aruco_dict, parameters=parameters) frame_markers = aruco.drawDetectedMarkers(frame.copy(), corners, ids) plt.figure() plt.imshow(frame_markers) for i in range(len(ids)): c = corners[i][0] plt.plot([c[:, 0].mean()], [c[:, 1].mean()], "o", label = "id={0}".format(ids[i])) plt.legend() plt.show() vid.release() # Destroy all the windows cv2.destroyAllWindows()  normal example for finding and marking aruco

Answer: API changed for 4.7.x, I have updated a small snippet. Now you need to instantiate ArucoDetector object. import cv2 as cv dictionary = cv.aruco.getPredefinedDictionary(cv.aruco.DICT_4X4_250) parameters = cv.aruco.DetectorParameters() detector = cv.aruco.ArucoDetector(dictionary, parameters) frame = cv.imread(...) markerCorners, markerIds, rejectedCandidates = detector.detectMarkers(frame)   OpenCV: cv::aruco::ArucoDetector Class Reference OpenCV: cv::aruco::Dictionary Class Reference
Question: Suppose we have the following code: class Base: a: int class Derived(Base): b: int print(Derived.__annotations__)  Running this script in all recent versions of python will print {'b': <class 'int'>} (that is, the class members we explicitly defined in Derived. In python 3.10, using inspect.get_annotations(Derived) will result in the same output. My question is, how do I get the type annotations for all the members of Derived? I'd like some function that outputs {'a': <class 'int'>, 'b': <class 'int'>}. This seems like a common enough use case that there should be a built-in function to do this, but I haven't found any. I'm running Python 3.10, so recently added functions are good for me.

Answer: This is a job for typing.get_type_hints, not raw annotation inspection: import typing full_hints = typing.get_type_hints(Derived)  This will also resolve string annotations, and recursively replace Annotated[T, ...] with T. Use cases that are interested in non-type-hint annotations can pass include_extras=True to get_type_hints to preserve Annotated annotations.
Question: I'm trying to convert very large integers to decimals, then convert those decimals to Fractions, and then convert the Fraction back to a decimal. I'm using the fractions and decimal packages to try and avoid floating point imprecision, however the accuracy still tapers off rather quickly. Is there any way to fix this / other ways of doing this?  import fractions import decimal def convert(exampleInt): power_of_10 = len(str(exampleInt)) decimal.getcontext().prec = 10000 exampleDecimal = decimal.Decimal(exampleInt) / (decimal.Decimal(10) ** power_of_10) exampleFraction = fractions.Fraction(str(exampleDecimal)).limit_denominator() backToDecimal = exampleFraction.numerator / decimal.Decimal(exampleFraction.denominator) print(f"backToDecimal: {backToDecimal}") convert(34163457536856478543908582348965743529867234957893246783427568734742390675934285342)  Which outputs: .341634575369123189552597490138153768602695082851231192155069838....

Answer: It's because of calling the limit_denominator(). Also it's quite inefficient to convert using an intermediate string. Convert a Decimal object into a Fraction object using the constructor like the following.(It's Mark Dickinson's solution.) import fractions import decimal decimal.getcontext().prec = 100 d = decimal.Decimal(34163457536856478543908582348965743529867234957893246783427568734742390675934285342) d = d / 10**(d.adjusted() + 1) f = fractions.Fraction(d) # This is equivalent to Fraction(*d.as_integer_ratio()) d2 = decimal.Decimal(f.numerator) / f.denominator assert(d2 == d)
Question: I am currently looking to speed up my code using the power of multiprocessing. However I am encountering some issues when it comes to calling the compiled code from python, as it seems that the compiled file disappears from the code's view when it includes any form of multiprocessing. For instance, with the following test code: #include <omp.h> int main() { int thread_id; #pragma omp parallel { thread_id = omp_get_thread_num(); } return 0; }  Here, I compile the program, then turn it into a .so file using the command gcc -fopenmp -o theories/test.so -shared -fPIC -O2 test.c  I then attempt to run the said code from test.py: from ctypes import CDLL import os absolute_path = os.path.dirname(os.path.abspath(__file__)) # imports the c libraries test_lib_path = absolute_path + '/theories/test.so' test = CDLL(test_lib_path) test.main() print('complete')  I get the following error: FileNotFoundError: Could not find module 'C:\[my path]\theories\test.so' (or one of its dependencies). Try using the full path with constructor syntax.  However, when I comment out the multiprocessing element to get the follwing code: #include <omp.h> int main() { int thread_id; /* #pragma omp parallel { thread_id = omp_get_thread_num(); } */ return 0; }  I then have a perfect execution with the python program printing out "complete" at the end. I'm wondering how this has come to happen, and how the code can seemingly be compiled fine but then throw problems only once it's called from python (also I have checked and the file is in fact created). UPDATES:  I have now checked that I have libgomp-1.dll installed I have uninstalled and reinstalled MinGW, with no change happening. I have installed a different, 64 bit version of gcc and, using a different (64 bit python 3.10) version of python have reproduced the same error. This also has libgomp-1.dll. 

Answer: I think this has the same root cause as [SO]: Can't import dll module in Python (@CristiFati's answer) (also check [SO]: PyWin32 and Python 3.8.0 (@CristiFati's answer)). A .dll (.so) is only loaded when its dependencies are successfully loaded (recursively). [SO]: Python Ctypes - loading dll throws OSError: [WinError 193] %1 is not a valid Win32 application (@CristiFati's answer) focuses on a different error, but covers this topic. When commenting out omp_get_thread_num call, the linker no longer links test.so to libgomp*.dll (as it doesn't need anything from it), and code runs fine (all required .dlls are found). To fix the issue, you should add to os.add_dll_directory (before attempting to load the .so):  libgomp*.dll's directory  MinGW's bin directory  Any other directory that may contain .dlls that are required (dependents)   To see a .dll dependencies, check [SO]: Discover missing module using command-line ("DLL load failed" error) (@CristiFati's answer). Notes:  main as an exported function from an so can be misleading  Although not required here (at least for the current code), check [SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer)
Question: I'm working with datetime information in pandas and wanted to convert a bunch of datetime64[ns] columns to str. I noticed a different behavior from the two approaches that I expected to yield the same result. Here's a MCVE. import pandas as pd # Create a dataframe with dates according to ISO8601 df = pd.DataFrame({"dt_column": ["2023-01-01", "2023-01-02", "2023-01-02"]}) # Convert the strings to datetimes # (I expect the time portion to be 00:00:00) df["dt_column"] = pd.to_datetime(df["dt_column"]) df["str_from_astype"] = df["dt_column"].astype(str) df["str_from_apply"] = df["dt_column"].apply(str) print(df) print() print("Datatypes of the dataframe") print(df.dtypes)  Output  dt_column str_from_astype str_from_apply 0 2023-01-01 2023-01-01 2023-01-01 00:00:00 1 2023-01-02 2023-01-02 2023-01-02 00:00:00 2 2023-01-02 2023-01-02 2023-01-02 00:00:00 Datatypes of the dataframe dt_column datetime64[ns] str_from_astype object str_from_apply object dtype: object  If I use .astype(str) the time information is lost and when I use .apply(str) the time information is retained (or inferred). Why is that? (Pandas v1.5.2, Python 3.9.15)

Answer: The time information is never lost, if you use 2023-01-02 12:00, you'll see that all times will be present with astype, but also visible in the original datetime column:  dt_column str_from_astype str_from_apply 0 2023-01-01 00:00:00 2023-01-01 00:00:00 2023-01-01 00:00:00 1 2023-01-02 00:00:00 2023-01-02 00:00:00 2023-01-02 00:00:00 2 2023-01-02 12:00:00 2023-01-02 12:00:00 2023-01-02 12:00:00  With apply, the python str builtin is applied on each Timestamp object, which always shows a full format: str(pd.Timestamp('2023-01-01')) # '2023-01-01 00:00:00'  With astype, the formatting is handled by pandas.io.formats.format.SeriesFormatter, which is a bit smarter and decides on the output format depending on the context (here other values in the Series and the presence of a non-null time). The canonical way to be explicit is anyway to use dt.strftime: # without time df["dt_column"].dt.strftime('%Y-%m-%d') # with time df["dt_column"].dt.strftime('%Y-%m-%d %H:%M:%S')
Question: I have a pyspark DataFrame like:  +------------------------+ | ids| +------------------------+ |[101826, 101827, 101576]| +------------------------+  and I want explode this dataframe like:  +------------------------+ | id| ids| +------------------------+ |101826 |[101827, 101576]| |101827 |[101826, 101576]| |101576 |[101826, 101827]| +------------------------+  How can I do using pyspark udf or other methods?

Answer: The easiest way out is to copy id into ids. Explode id and use array except to exclude each id in the row. Code below.  ( df1.withColumn('ids', col('id')) .withColumn('id',explode('id')) .withColumn('ids',array_except(col('ids'), array('id'))) ).show(truncate=False) +------+----------------+ |id |ids | +------+----------------+ |101826|[101827, 101576]| |101827|[101826, 101576]| |101576|[101826, 101827]| +------+----------------+
Question: I have 2 images, test1.jpg and test2.jpg that are RGB images. They have been converted from a 2D numpy array so they are monochrome images. They have the same shape. When I use the paste function, I only see one of the images instead of both. Here are the test1 and test2 jpgs:  . This is what I get after doing test1.paste(test2) and test1.save('final.jpg'):  Why is it only showing test2? Here is my code: im1 = Image.open('test1.jpg') im2 = Image.open('test2.jpg') im1.paste(im2) im1.save('final.jpg') 

Answer: You simply need to choose the lighter of your two images at each point with PIL Channel Operations: from PIL import Image, ImageChops im1 = Image.open('test1.jpeg') im2 = Image.open('test2.jpeg') # Choose lighter of the two images at each pixel location combined = ImageChops.lighter(im1,im2)    Note that you could use paste() as you originally intended, but that it will paste all the black as well as the white pixels from image2 over image1. In order to avoid that, you would need to make a mask and only paste where image2 is non-zero. That might look like this: im1 = Image.open('test1.jpeg') im2 = Image.open('test2.jpeg') # Make greyscale mask from image2 mask = im2.convert('L') mask = mask.point(lambda i: 255 if i>0 else 0) # Paste image2 into image1 only where image2 has non-black content im1.paste(im2, mask=mask)  I just think the ImageChops.lighter() method is simpler.  Note that these two methods will give subtly different results. For example, if a pixel is 192 in image1 and 67 in image2, the ImageChops.lighter() method will result in 192, whereas the paste() method will see there is something in image2, and therefore give you the 67. Your choice!
Question: I am trying to read the window.appCache from a glassdoor reviews site. url = "https://www.glassdoor.com/Reviews/Alteryx-Reviews-E351220.htm" html = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}) soup = BeautifulSoup(html.content,'html.parser') text = soup.findAll("script")[0].text  This isolates the dict I need however when I tried to do json.loads() I get the following error: raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)  I checked the type of text and it is str. When I print text to a file, it looks something like this (just a snippet as the output is about 5000 lines): window.appCache={"appName":"reviews","appVersion":"7.14.12","initialState" {"surveyEndpoint":"https:\u002F\u002Femployee-pulse-survey-b2c.us-east-1.prod.jagundi.com", "i18nStrings":{"_":"JSON MESSAGE BUNDLE - do not remove", "eiHeader.seeAllPhotos":" See All Photos","eiHeader.viewJobs":"View Jobs", "eiHeader.bptw.description":"This employer is a winner of the [year] Best Places to Work award. Winners were determined by the people who know these companies best...  I am only concerned with the "reviews":[ field that is buried about halfway through the data, but I can't seem to parse the string into json and retrieve what I need.

Answer: One solution is to parse the required data with re/json module: import json import pprint import re import requests url = "https://www.glassdoor.com/Reviews/Alteryx-Reviews-E351220.htm" html = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).text reviews = re.search(r'"reviews":(\[.*?}\])}', html, flags=re.S).group(1) reviews = json.loads(reviews) pprint.pprint(reviews)  Prints: [{'__typename': 'EmployerReview', 'advice': "Don't rush too finish a project", 'adviceOriginal': None, 'cons': 'Typical like other companies where newbies get higher salary and ' 'you have to work your way up for promotions nothing really bad', 'consOriginal': None, 'countHelpful': 0, 'countNotHelpful': 0, 'divisionLink': None, 'divisionName': None, 'employer': {'__ref': 'Employer:351220'}, 'employerResponses': [], 'employmentStatus': None, 'isCovid19': False, 'isCurrentJob': True, 'isLanguageMismatch': False, 'isLegal': True, 'jobEndingYear': None, 'jobTitle': None, 'languageId': 'eng', 'lengthOfEmployment': 6, 'location': None, ...and so on.
Question: I have this code and I don't know how to make the red cube disappear after 2 seconds. import pygame import sys from pygame.locals import * pygame.init() a=0 #display prozor = pygame.display.set_mode((800,800)) FPS = pygame.time.Clock() FPS.tick(60) #boje green=pygame.Color(0 ,255 , 0) red=pygame.Color(255, 0, 0) yellow=pygame.Color(255, 255, 0) blue=pygame.Color(0, 0, 255) black=pygame.Color(0, 0, 0) white=pygame.Color(255, 255, 255) #class class Cube: def update(self): self.cx, self.cy = pygame.mouse.get_pos() self.square = pygame.Rect(self.cx, self.cy, 50, 50) def draw(self): pygame.draw.rect(prozor, (255, 0, 0), self.square) cube = Cube() drawing_cube = False #objekt for j in range(16): for i in range(800): if i%50 == 0: pygame.draw.rect(prozor, green, ((0+i, a), (50, 50),),1) a=a+50 #gameloop while True: for event in pygame.event.get(): if event.type == QUIT: pygame.quit() sys.exit if event.type == pygame.MOUSEBUTTONDOWN: cube.update() drawing_cube = True if drawing_cube: cube.draw() pygame.display.flip() pygame.display.update()  Adding it to the draw class will just freeze the game and shut it. def draw(self): pygame.draw.rect(prozor, red, self.square) time.sleep(3) pygame.draw.rect(prozor, black, self.square  I tried making another class called delete which would delete the cube after 3 seconds using the time module. def delete(self): time.sleep(3) pygame.draw.rect(prozor, black, self.square)  and added it here while True: for event in pygame.event.get(): if event.type == QUIT: pygame.quit() sys.exit if event.type == pygame.MOUSEBUTTONDOWN: cube.update() drawing_cube = True if drawing_cube: cube.draw() pygame.display.flip() **cube.delete** pygame.display.flip() pygame.display.update()  but the cube is not disappearing.

Answer: Use pygame.time.get_ticks to measure the time in milliseconds. Calculate the time when the cube must disappear again and hide the cube if the current time is greater than the calculated time. You also need to clear the display (prozor.fill(0)) and redraw the scene in each frame: drawing_cube = False hide_cube_time = 0 clock = pygame.time.Clock() run = True while run: clock.tick(100) current_time = pygame.time.get_ticks() for event in pygame.event.get(): if event.type == QUIT: run = False if event.type == pygame.MOUSEBUTTONDOWN: cube.update() drawing_cube = True hide_cube_time = current_time + 2000 # 2000 milliseconds == 2 sconds if current_time > hide_cube_time: drawing_cube = False prozor.fill(0) a=0 for j in range(16): for i in range(800): if i%50 == 0: pygame.draw.rect(prozor, green, (0+i, a, 50, 50), 1) a=a+50 if drawing_cube: cube.draw() pygame.display.update() pygame.quit() sys.exit()  Note, the typical PyGame application loop has to:  limit the frames per second to limit CPU usage with pygame.time.Clock.tick handle the events by calling either pygame.event.pump() or pygame.event.get(). update the game states and positions of objects dependent on the input events and time (respectively frames) clear the entire display or draw the background draw the entire scene (blit all the objects) update the display by calling either pygame.display.update() or pygame.display.flip()
Question: faced with such an error: Local cdn resources have problems on chrome/safari when used in jupyter-notebook. It appears when working with the pyvis library. net = Network(notebook=True) net.add_nodes( [1, 2, 3, 4, 5], # node ids label=['Node #1', 'Node #2', 'Node #3', 'Node #4', 'Node #5'], # node labels # node titles (display on mouse hover) title=['Main node', 'Just node', 'Just node', 'Just node', 'Node with self-loop'], color=['#d47415', '#22b512', '#42adf5', '#4a21b0', '#e627a9'] # node colors (HEX) ) net.add_edges([(1, 2), (1, 3), (2, 3), (2, 4), (3, 5), (5, 1)]) net.show('graph.html')   I tried switching the browser to dataspell

Answer: From pyvis documentation:  while using notebook in chrome browser, to render the graph, pass additional kwarg ‘cdn_resources’ as ‘remote’ or ‘inline’  I did net = Network(notebook=True, cdn_resources='in_line') A note from me - you have to use 'in_line' instead of 'inline'. From sources: assert cdn_resources in ["local", "in_line", "remote"], "cdn_resources not in [local, in_line, remote]."  Additionally, pyvis output render is an open feature request for DataSpell at the moment. Watch and upvote this ticket if you're interested: DS-3446
Question: When I run this code: from telegram.ext import * import keys print('Starting a bot....') def start_commmand(update, context): update.message.reply_text('Hello! Welcome To Store!') if __name__ == '__main__': updater = Updater(keys.token, True) dp = updater.dispatcher # Commands dp.add.handler(CommandHandler('start', start_commmand)) # Run bot updater.start_polling(1.0) updater.idle()  I get this error: Traceback (most recent call last): File "C:\Users\pc\PycharmProjects\telegram\main.py", line 11, in <module> dp = updater.dispatcher AttributeError: 'Updater' object has no attribute 'dispatcher'  I attempted to resolve this issue by updating the library but the error remained.

Answer: You probably found an example for v13, but since a few days the v20 for python-telegram-bot is out. Now you have to build your application differently and you have to use async functions. This should work: from telegram.ext import * import keys print('Starting a bot....') async def start_commmand(update, context): await update.message.reply_text('Hello! Welcome To Store!') if __name__ == '__main__': application = Application.builder().token(keys.token).build() # Commands application.add_handler(CommandHandler('start', start_commmand)) # Run bot application.run_polling(1.0)  Also, here are some good examples for the python-telegram-bot library.