eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to concat multiple Pandas DataFrame columns with different tokens. For example, my dataset looks like this : dataframe = pd.DataFrame({'col_1' : ['aaa','bbb','ccc','ddd'], 'col_2' : ['name_aaa','name_bbb','name_ccc','name_ddd'], 'col_3' : ['job_aaa','job_bbb','job_ccc','job_ddd']})  I want to output something like this:  features 0 aaa <0> name_aaa <1> job_aaa 1 bbb <0> name_bbb <1> job_bbb 2 ccc <0> name_ccc <1> job_ccc 3 ddd <0> name_ddd <1> job_ddd  Explanation : concat each column with "<{}>" where {} will be increasing numbers. What I've tried so far: I don't want to modify original DataFrame so I created two new dataframe: features_df = pd.DataFrame() final_df = pd.DataFrame() for iters in range(len(dataframe.columns)): features_df[dataframe.columns[iters]] = dataframe[dataframe.columns[iters]] + ' ' + "<{}>".format(iters) final_df['features'] = features_df[features_df.columns].agg(' '.join, axis=1)  There is an issue I am facing, It's adding <2> at last but I want output like above, also this is not panda's way to do this task, How I can make it more efficient?

Answer: from itertools import chain dataframe['features'] = dataframe.apply(lambda x: ''.join([*chain.from_iterable((v, f' <{i}> ') for i, v in enumerate(x))][:-1]), axis=1) print(dataframe)  Prints:  col_1 col_2 col_3 features 0 aaa name_aaa job_aaa aaa <0> name_aaa <1> job_aaa 1 bbb name_bbb job_bbb bbb <0> name_bbb <1> job_bbb 2 ccc name_ccc job_ccc ccc <0> name_ccc <1> job_ccc 3 ddd name_ddd job_ddd ddd <0> name_ddd <1> job_ddd
Question: I am using tensorflow V2.2 and run into TyepError when I do model.evaluate. Can someone advise what the issues may be? A screenshot of the execution and error message is shown below. 

Answer: you need to define a metric when you compile the model model.compile('adam', 'binary_crossentropy', metrics='accuracy') in this way during evaluation, loss and accuracy are returned
Question: I have a picture were I want to change all white-ish pixels to grey, but only for a certain area of the image. Example picture, I just want to change the picture outside of the red rectangle, without changing the image within the red rectangle:  I already have the general code, which was part of someone elses Stackoverflow question, that changes the colour of every white pixel instead of only just the one outside of an area. image = cv.imread("meme 2.jpg") hsv = cv.cvtColor(image, cv.COLOR_BGR2HSV) # Define lower and uppper limits of what we call "white-ish" sensitivity = 19 lower_white = np.array([0, 0, 255 - sensitivity]) upper_white = np.array([255, sensitivity, 255]) # Mask image to only select white mask = cv.inRange(hsv, lower_white, upper_white) # Change image to grey where we found brown image[mask > 0] = (170, 170, 170) cv.imwrite(file, image) 

Answer: Here is one way to do that in Python/OpenCV.  Read the input Convert to HSV color space Threshold on desired color to make a mask Use the mask to change the color of all corresponding pixels in the image Draw a new rectangular mask for the region where you do not want to change Invert the new mask for the region where you do want to change Apply the new mask to the original image Apply the inverted new mask to the color changed image Add the two results together to form the final image Save the results   Input:  import cv2 import numpy as np # Read image image = cv2.imread('4animals.jpg') # Convert to HSV hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) # Define lower and uppper limits of what we call "white-ish" sensitivity = 19 lower_white = np.array([0, 0, 255 - sensitivity]) upper_white = np.array([255, sensitivity, 255]) # Create mask to only select white mask = cv2.inRange(hsv, lower_white, upper_white) # Change image to grey where we found white image2 = image.copy() image2[mask > 0] = (170, 170, 170) # Create new rectangular mask that is white on black background x,y,w,h = 33,100,430,550 mask2 = np.zeros_like(image) cv2.rectangle(mask2, (x,y), (x+w,y+h), (255, 255, 255), -1) # invert mask mask2_inv = 255 - mask2 # apply mask to image image_masked = cv2.bitwise_and(image, mask2) # apply inverted mask to image2 image2_masked = cv2.bitwise_and(image2, mask2_inv) # add together result = cv2.add(image_masked, image2_masked) # save results cv2.imwrite('4animals_mask.jpg', mask) cv2.imwrite('4animals_modified.png', image2) cv2.imwrite('4animals_mask2.jpg', mask2) cv2.imwrite('4animals_mask2_inv.jpg', mask2_inv) cv2.imwrite('4animals_masked.jpg', image_masked) cv2.imwrite('4animals_modified_masked.jpg', image2_masked) cv2.imwrite('4animals_result.jpg', result) cv2.imshow('mask', mask) cv2.imshow('image2', image2) cv2.imshow('mask2', mask2 ) cv2.imshow('mask2_inv', mask2_inv) cv2.imshow('image_masked', image_masked) cv2.imshow('image2_masked', image2_masked) cv2.imshow('result', result) cv2.waitKey(0) cv2.destroyAllWindows()   Color mask:  Rectangle mask:  Inverted rectangle mask:  Color changed image:  Masked input:  Masked color changed image:  Result:
Question: See this answer.  I think your confusion is that you're mixing up the concept of string literals in source code with actual string values.  What is the difference between string literals and string values? I did not understand this.

Answer: A string literal is a piece of text you can write in your program's source code, beginning and ending with quotation marks, that tells Python to create a string with certain contents. It looks like 'asdf'  or ''' multiline content '''  or 'the thing at the end of this one is a line break\n'  In a string literal (except for raw string literals), special sequences of characters known as escape sequences in the string literal are replaced with different characters in the actual string. For example, the escape sequence \n in a string literal is replaced with a line feed character in the actual string. Escape sequences begin with a backslash.  A string is a Python object representing a text value. It can be built from a string literal, or it could be read from a file, or it could originate from many other sources. Backslashes in a string have no special meaning, and backslashes in most possible sources of strings have no special meaning either. For example, if you have a file with backslashes in it, looking like this: asdf\n  and you do with open('that_file.txt') as f: text = f.read()  the \n in the file will not be replaced by a line break. Backslashes are special in string literals, but not in most other contexts.  When you ask for the repr representation of a string, either by calling repr or by displaying the string interactively: >>> some_string = "asdf" >>> some_string 'asdf'  Python will build a new string whose contents are a string literal that would evaluate to the original string. In this example, some_string does not have ' or " characters in it. The contents of the string are the four characters asdf, the characters displayed if you print the string: >>> print(some_string) asdf  However, the repr representation has ' characters in it, because 'asdf' is a string literal that would evaluate to the string. Note that 'asdf' is not the same string literal as the "asdf" we originally used - many different string literals can evaluate to equal strings.
Question: I am new and I try to grap source code of an Web page for tutorial.I got beautifulsoup install,request install. At first I want to grap the source.I am doing this scraping job from "https://pythonhow.com/example.html".I am not doing anything illegal and I think this site also established for this purposes.Here's my code: import requests from bs4 import BeautifulSoup r=requests.get("http://pythonhow.com/example.html") c=r.content c  And i got the mod security error: b'<head><title>Not Acceptable!</title></head><body><h1>Not Acceptable!</h1><p>An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.</p></body></html>'  Thanks for all who re dealing with me.Respectly

Answer: You can easily fix this issue by providing a user agent to the request. By doing so, the website will think that someone is actually visiting the site using a web browser.  Here is the code that you want to use: import requests from bs4 import BeautifulSoup headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:55.0) Gecko/20100101 Firefox/55.0', } r = requests.get("http://pythonhow.com/example.html", headers=headers) c = r.content print(c)  Which gives you the expected output b'<!DOCTYPE html>\n<html>\n<head>\n<style>\ndiv.cities {\n background-color:black;\n color:white;\n margin:20px;\n padding:20px;\n} \n</style>\n</head>\n<body>\n<h1 align="center"> Here are three big cities </h1>\n<div class="cities">\n<h2>London</h2>\n<p>London is the capital of England and it\'s been a British settlement since 2000 years ago. </p>\n</div>\n<div class="cities">\n<h2>Paris</h2>\n<p>Paris is the capital city of France. It was declared capital since 508.</p>\n</div>\n<div class="cities">\n<h2>Tokyo</h2>\n<p>Tokyo is the capital of Japan and one of the most populated cities in the world.</p>\n</div>\n</body>\n</html>'
Question: I am trying to implement the python logging using TimedRotatingFileHandler i'm getting the problem in adding the file extension in log filename here is my code  Path(".\\Log").mkdir(parents=True, exist_ok=True) LOGGING_MSG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s' LOGGING_DATE_FORMAT = '%m-%d %H:%M:%S' formatter = logging.Formatter(LOGGING_MSG_FORMAT, LOGGING_DATE_FORMAT) handler = TimedRotatingFileHandler(".\\Log\\info.log",'midnight',1) handler.setFormatter(formatter) handler.setLevel(logging.INFO) root_logger = logging.getLogger() root_logger.addHandler(handler)  using this code very first time i'm getting the fileName "info.log" as expected, but when it rolls over to midnight the fileName i'm getting is "info.log.2020-05-22" but what i'm expecting is "info.2020-05-22.log". How i can append the handler suffix before to file extension(.log)?

Answer: You should use a custom namer: handler.namer = lambda name: name + ".log"  Unfortunately, the namer function gets the processed name. The name param would be like "info.log.2020-05-22", so you'll end up with "info.log.2020-05-22.log". If double .log is not acceptable just remove the initial one: handler.namer = lambda name: name.replace(".log", "") + ".log"
Question: I'm a new Python programmer using the Mac version of VS Code 1.45.1 to create a Django project. I have the Python and Django extensions installed. Every time I save a Django file, Code pops up this window: Formatter autopep8 is not installed. Install? Source: Python (Extension) [Yes] [Use black] [Use yapf]  I keep clicking the "Yes" button to install the autopep8 extension but this message keeps popping up nevertheless. Is there some trick to configuring VS Code so that this extension will be installed permanently and I stop getting this error?

Answer: You will receive this prompt if  You have "formatOnSave" turned on as a setting You selected autopep8 as your formatter The Python extension can't find autopep8  So the options are:  Turn off formatting on save Make sure you successfully installed autopep8 into your environment or you specified the path to autopep8 in your settings  My guess is there's an installation failure because you are using a globally installed interpreter and you're not allowed to install where pip wants to put autopep8.
Question: I have to calculate the distance on a hilbert-curve from 2D-Coordinates. With the hilbertcurve-package i built my own "hilbert"-function, to do so. The coordinates are stored in a dataframe (col_1 and col_2). As you see, my function works when applied to two values (test). However it just does not work when applied row wise via apply-function! Why is this? what am I doing wrong here? I need an additional column "hilbert" with the hilbert-distances from the x- and y-coordinate given in columns "col_1" and "col_2". import pandas as pd from hilbertcurve.hilbertcurve import HilbertCurve df = pd.DataFrame({'ID': ['1', '2', '3'], 'col_1': [0, 2, 3], 'col_2': [1, 4, 5]}) def hilbert(x, y): n = 2 p = 7 hilcur = HilbertCurve(p, n) dist = hilcur.distance_from_coordinates([x, y]) return dist test = hilbert(df.col_1[2], df.col_2[2]) df["hilbert"] = df.apply(hilbert(df.col_1, df.col_2), axis=0)  The last command ends in error: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().  Thank you for your help!

Answer: Since you have hilbert(df.col_1, df.col_2) in the apply, that's immediately trying to call your function with the full pd.Serieses for those two columns, triggering that error. What you should be doing is: df.apply(lambda x: hilbert(x['col_1'], x['col_2']), axis=1)  so that the lambda function given will be applied to each row.
Question: I just installed poetry with the following install script curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python3  However, when I execute poetry it fails with the following error $ poetry /usr/bin/env: ‘python’: No such file or directory  I recently upgraded to ubuntu 20.04, is this an issue with the upgrade or with poetry?

Answer: poetry is dependent on whatever python is and doesn't attempt to use a specific version of python unless otherwise specified. The above issue will exist on ubuntu systems moving forward 20.04 onwards as python2.7 is deprecated and the python command does not map to python3.x You'll find specifying an alias for python to python3 won't work ( unless, perhaps you specify this in your bashrc instead of any other shell run command file ) as poetry spins it's own shell to execute commands. Install the following package instead sudo apt install python-is-python3  It should be noted that you can install python2.7 if you want to and poetry should run fine.
Question: I am not really sure how to explain this but I have 2 polygons, Polygon1 and Polygon2. These polygons overlapped with each other. How to do I get Polygon2 using Shapely without the P from Polygon1. 

Answer: You are looking for a difference. In Shapely you can calculate it either by using a difference method or by simply subtracting* one polygon from another: from shapely.geometry import Polygon polygon1 = Polygon([(0.5, -0.866025), (1, 0), (0.5, 0.866025), (-0.5, 0.866025), (-1, 0), (-0.5, -0.866025)]) polygon2 = Polygon([(1, -0.866025), (1.866025, 0), (1, 0.866025), (0.133975, 0)])   difference = polygon2.difference(polygon1) # or difference = polygon2 - polygon1   See docs for more set-theoretic methods.  *This feature is not documented. See issue on GitHub: Document set-like properties.
Question: I have a pandas dataframe df which I want to overwrite to a sheet Data of an excel file while preserving all the other sheets since other sheets have formulas linked to sheet Data I used the following code but it does not overwrite an existing sheet, it just creates a new sheet with the name Data 1 with pd.ExcelWriter(filename, engine="openpyxl", mode="a") as writer: df.to_excel(writer, sheet_name="Data")  Is there a way to overwrite on an existing sheet?

Answer: You can do it using openpyxl: import pandas as pd from openpyxl import load_workbook book = load_workbook(filename) writer = pd.ExcelWriter(filename, engine='openpyxl') writer.book = book writer.sheets = dict((ws.title, ws) for ws in book.worksheets) df.to_excel(writer, "Data") writer.save()  You need to initialize writer.sheets in order to let ExcelWriter know about the sheets. Otherwise, it will create a new sheet with the name that you pass.
Question: I've built a very simple TensorFlow Keras model with a single dense layer. It works perfectly fine outside a GradientTape block, but inside a GradientTape block it raises LookupError: No gradient defined for operation 'IteratorGetNext' (op type: IteratorGetNext) Code to reproduce: from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense import tensorflow as tf import numpy as np print(tf.__version__) model = Sequential() model.add(Dense(1, input_shape=(16,))) fake_data = np.random.random((1, 16)) print(model.predict(fake_data).shape) # works with tf.GradientTape() as tape: print(model.predict(fake_data).shape) # LookupError: No gradient defined for operation 'IteratorGetNext' (op type: IteratorGetNext)  This appears to work in TensorFlow 2.0.0, however it fails in TensorFlow 2.1.0 and 2.2.0 Here is a notebook that replicates the issue.

Answer: try to redefine the predict operation in GradientTape in this way with tf.GradientTape() as tape: print(model(fake_data).shape)
Question: I need to compute AB⁻¹ in Python / Numpy for two matrices A and B (B being square, of course). I know that np.linalg.inv() would allow me to compute B⁻¹, which I can then multiply with A. I also know that B⁻¹A is actually better computed with np.linalg.solve(). Inspired by that, I decided to rewrite AB⁻¹ in terms of np.linalg.solve(). I got to a formula, based on the identity (AB)ᵀ = BᵀAᵀ, which uses np.linalg.solve() and .transpose(): np.linalg.solve(a.transpose(), b.transpose()).transpose()  that seems to be doing the job: import numpy as np n, m = 4, 2 np.random.seed(0) a = np.random.random((n, n)) b = np.random.random((m, n)) print(np.matmul(b, np.linalg.inv(a))) # [[ 2.87169378 -0.04207382 -1.10553758 -0.83200471] # [-1.08733434 1.00110176 0.79683577 0.67487591]] print(np.linalg.solve(a.transpose(), b.transpose()).transpose()) # [[ 2.87169378 -0.04207382 -1.10553758 -0.83200471] # [-1.08733434 1.00110176 0.79683577 0.67487591]] print(np.all(np.isclose(np.matmul(b, np.linalg.inv(a)), np.linalg.solve(a.transpose(), b.transpose()).transpose()))) # True  and also comes up much faster for sufficiently large inputs: n, m = 400, 200 np.random.seed(0) a = np.random.random((n, n)) b = np.random.random((m, n)) print(np.all(np.isclose(np.matmul(b, np.linalg.inv(a)), np.linalg.solve(a.transpose(), b.transpose()).transpose()))) # True %timeit np.matmul(b, np.linalg.inv(a)) # 100 loops, best of 3: 13.3 ms per loop %timeit np.linalg.solve(a.transpose(), b.transpose()).transpose() # 100 loops, best of 3: 7.71 ms per loop   My question is: does this identity always stand correct or there are some corner cases I am overlooking?

Answer: In general, np.linalg.solve(B, A) is equivalent to B-1A. The rest is just math. In all cases, (AB)T = BTAT: https://math.stackexchange.com/q/1440305/295281. Not necessary for this case, but for invertible matrices, (AB)-1 = B-1A-1: https://math.stackexchange.com/q/688339/295281. For an invertible matrix, it is also the case that (A-1)T = (AT)-1: https://math.stackexchange.com/q/340233/295281. From that it follows that (AB-1)T = (B-1)TAT = (BT)-1AT. As long as B is invertible, you should have no issues with the transformation you propose in any case.
Question: I have a list called corpus that I am attempting TF-IDF on, using the sklearn in-built function. The list has 5 items. Each of these items comes from text files. I have generated a toy list called corpus for this example.  corpus = ['Hi what are you accepting here do you accept me', 'What are you thinking about getting today', 'Give me your password to get accepted into this school', 'The man went to the tree to get his sword back', 'go away to a far away place in a foreign land'] vectorizer = TfidfVectorizer(stop_words='english') vecs = vectorizer.fit_transform(corpus) feature_names = vectorizer.get_feature_names() dense = vecs.todense() lst1 = dense.tolist() df = pd.DataFrame(lst1, columns=feature_names) df  Using the above code, I was able to get a dataframe with 5 rows (for each item in the list) and n-columns with the tf-idf for each term in this corpus.  As a next step, I want to build the word cloud with largest tf-idf terms across the 5 items in the corpus getting the highest weight.  I tried the following:  x = vectorizer.vocabulary_ Cloud = WordCloud(background_color="white", max_words=50).generate_from_frequencies(x)  This clearly does not work. The dictionary is a list of words with an index attached to it, not a word scoring.  Hence, I need a dictionary that assigns the TF-IDF score to each word across the corpus. Then, the word cloud generated has the highest scored words as the largest size. 

Answer: You're almost there. You need to transpose to get the frequencies per term rather than term frequencies per document, then sum hem, then pass that series directly to your wordcloud df.T.sum(axis=1) accept 0.577350 accepted 0.577350 accepting 0.577350 away 0.707107 far 0.353553 foreign 0.353553 getting 0.577350 hi 0.577350 land 0.353553 man 0.500000 password 0.577350 place 0.353553 school 0.577350 sword 0.500000 thinking 0.577350 today 0.577350 tree 0.500000 went 0.500000 Cloud = WordCloud(background_color="white", max_words=50).generate_from_frequencies(df.T.sum(axis=1))
Question: I have installed pytorch with conda and transformers with pip. I can import transformers without a problem but when I try to import pipeline from transformers I get an exception: from transformers import pipeline --------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-4-69a9fd07ccac> in <module> ----> 1 from transformers import pipeline ImportError: cannot import name 'pipeline' from 'transformers' (C:\Users\Alienware\Anaconda3\envs\tf2\lib\site-packages\transformers\__init__.py)  This is a view of the directory where it searches for the init.py file:  What is causing the problem and how can I resolve it?

Answer: Check transformers version. Make sure you are on latest. Pipelines were introduced quite recently, you may have older version.
Question: First of all, i need to warn you: I'm new to asyncio, and i h I warn you right away, I'm new to asyncio, and I can hardly imagine what is in the library under the hood. Here is my code: import asyncio semaphore = asyncio.Semaphore(50) async def work(value): async with semaphore: print(value) await asyncio.sleep(10) async def main(): tasks = [] for i in range(0, 10000): tasks.append(asyncio.wait_for(work(i), timeout=3)) await asyncio.gather(*tasks) loop = asyncio.get_event_loop() future = asyncio.ensure_future(main()) loop.run_until_complete(future)  What i need: Coroutine work() to be completed for no more than 3 seconds, and no more than 50 pieces at a same time. After 3 seconds (timeout), the coroutine work() must stop execution, and a new 50 tasks must start to work. But in my case, after 3 seconds crashes: Traceback (most recent call last): File "C:/Users/root/PycharmProjects/LogParser/ssh/async/asyn_test.py", line 19, in <module> loop.run_until_complete(future) File "C:\Code\Python3\lib\asyncio\base_events.py", line 579, in run_until_complete return future.result() File "C:/Users/root/PycharmProjects/LogParser/ssh/async/asyn_test.py", line 15, in main await asyncio.gather(*tasks) File "C:\Code\Python3\lib\asyncio\tasks.py", line 449, in wait_for raise futures.TimeoutError() concurrent.futures._base.TimeoutError  Whatever I had not tried to catch this exception, no matter how many tasks remained, program breaks down. I need, upon reaching timeout, the program continued to the next tasks Please, teach me, how do I need properly implement this? Python 3.7, asyncio 3.4.3

Answer: You need to handle the exception. If you just pass it to gather, it will re-raise it. For example, you can create a new coroutine with the appropriate try/except: semaphore = asyncio.Semaphore(50) async def work(value): print(value) await asyncio.sleep(10) async def work_with_timeout(value): async with semaphore: try: return await asyncio.wait_for(work(value), timeout=3) except asyncio.TimeoutError: return None async def main(): tasks = [] for i in range(0, 10000): tasks.append(work_with_timeout(i)) await asyncio.gather(*tasks)