eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm attempting to create an isolated virtual environment running tensorflow & tf2onnx using a jupyter notebook in vscode. The tf2onnx packge recommends python 3.7, and my local 3.7.9 version usually works well with tensorflow projects, so I have local and global versions set to 3.7.9 using pyenv. The following is my setup procedure: python -m venv .venv Then after starting a new terminal in vscode: pip install tensorflow==2.7.0 pip freeze > requirements.txt After this, in a cell in my jupyter notebook, the following line fails import tensorflow.keras as keras  Exception:  TypeError: Descriptors cannot not be created directly. If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. If you cannot immediately regenerate your protos, some other possible workarounds are: 1. Downgrade the protobuf package to 3.20.x or lower. 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).   At this point, the protobuf package version is showing as v4.21.0 in my requirements file. I've attempted to pre-install the 3.20.1 version into the virtual environment before installing tensorflow but this yields no effect. Here is the full requirements file after installing tensorflow: absl-py==1.0.0 astunparse==1.6.3 cachetools==5.1.0 certifi==2022.5.18.1 charset-normalizer==2.0.12 flatbuffers==2.0 gast==0.4.0 google-auth==2.6.6 google-auth-oauthlib==0.4.6 google-pasta==0.2.0 grpcio==1.46.3 h5py==3.7.0 idna==3.3 importlib-metadata==4.11.4 keras==2.7.0 Keras-Preprocessing==1.1.2 libclang==14.0.1 Markdown==3.3.7 numpy==1.21.6 oauthlib==3.2.0 opt-einsum==3.3.0 protobuf==4.21.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 requests==2.27.1 requests-oauthlib==1.3.1 rsa==4.8 six==1.16.0 tensorboard==2.9.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorflow==2.7.0 tensorflow-estimator==2.7.0 tensorflow-io-gcs-filesystem==0.26.0 termcolor==1.1.0 typing-extensions==4.2.0 urllib3==1.26.9 Werkzeug==2.1.2 wrapt==1.14.1 zipp==3.8.0 

Answer: A recent change in protobuf is causing TensorFlow to break. Downgrading before installing TensorFlow might not work because TensorFlow might be bumping up the version itself. Check if that is what happens during the installation. You might want to either: Downgrade with pip install --upgrade "protobuf<=3.20.1"  after installing TensorFlow, or Upgrade TensorFlow to the latest version, as TensorFlow has updated their setup file in their 2.9.1 release.
Question: from fastapi import FastAPI from fastapi.params import Body app = FastAPI() @app.post("/createposts") def create_posts(payload: dict = Body(...)): print(payload) return {"new_post" : f"title {payload["title"]} content: {payload["content"]}"}  I'm trying to create an API with Fastapi, but every time I run the code I get this error related to the return statement: SyntaxError: f-string: unmatched '[' Thank you!

Answer: Please change return {"new_post" : f"title {payload["title"]} content: {payload["content"]}"}  to return {"new_post" : f"title {payload['title']} content: {payload['content']}"} You can't have " quotes inside f"..." The error says that after the first [ the string stops and breaks.
Question: I have students names, scores in different subjects, subjects names. I want to add a column to the data frame, which contains the subject in which each student had the highest score. Here is the data: Input data would be:  Output data (the result data frame) would be :  My try at this (didn't work obviously): Data['Subject with highest score'] = Data.groupby(['Names','Subject'])[['Scores']].transform(lambda x: x.max()) 

Answer: Sort the values by Scores, then group the dataframe by Names and transform the column Subject with last df['S(max)'] = df.sort_values('Scores').groupby('Names')['Subject'].transform('last')  Alternatively, we can group the dataframe by Names then transform Scores with idxmax to broadcast the indices corresponding to row having max Score, then use those indices to get the corresponding rows from Subject column df['S(max)'] = df.loc[df.groupby('Names')['Scores'].transform('idxmax'), 'Subject'].tolist()    Names Scores Subject S(max) 0 Dan 98 Math Math 1 Dan 88 English Math 2 Dan 90 Biology Math 3 Bob 80 Math Chemistry 4 Bob 93 Chemistry Chemistry 5 Bob 70 Sports Chemistry 6 Bob 85 French Chemistry 7 Michael 100 History History 8 Sandra 67 French French 9 Michael 89 Math History 10 Michael 74 Sports History 11 Jacky 65 Biology Physics 12 Jacky 100 Physics Physics 13 Jacky 90 Geometry Physics 14 Jacky 87 Geography Physics 15 Jacky 69 Math Physics 16 Dan 73 Sports Math 17 Sandra 50 History French
Question: Each time that I initiate a new webdriver the following text is written to the console: [WDM] - ====== WebDriver manager ====== [WDM] - Current google-chrome version is 102.0.5005 [WDM] - Get LATEST chromedriver version for 102.0.5005 google-chrome [WDM] - Driver [C:\Users\klaas\.wdm\drivers\chromedriver\win32\102.0.5005.61\chromedriver.exe] found in cache  My goal was to stop selenium from printing this message to the console. Stack Overflow threads with similar topics to this one showed two options that did not work for me. The first one is: from selenium.webdriver.chrome.options import Options options = Options() options.add_experimental_option("excludeSwitches", ["enable-logging"]) driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)  and the second one is: import logging from selenium.webdriver.remote.remote_connection import LOGGER LOGGER.setLevel(logging.WARNING)  Both of these solutions did work for some others but not for me. Is there some other way to stop selenium from printing webdriver messages? Solution: As suggested by MohitC the following code prevented the webdriver-manager messages to be printed: import logging logging.getLogger('WDM').setLevel(logging.NOTSET) 

Answer: These are webdriver-manager logs. You can either uninstall it if you are not using or disable logging as below import os os.environ['WDM_LOG'] = "false"  You can also try import logging logging.getLogger('WDM').setLevel(logging.NOTSET)
Question: I am trying to figure out how to get value_counts from how many times a specific text value is listed in the column. Example data: d = {'Title': ['Crash Landing on You', 'Memories of the Alhambra', 'The Heirs', 'While You Were Sleeping', 'Something in the Rain', 'Uncontrollably Fond'], 'Cast' : ['Hyun Bin,Son Ye Jin,Seo Ji Hye', 'Hyun Bin,Park Shin Hye,Park Hoon', 'Lee Min Ho,Park Shin Hye,Kim Woo Bin', 'Bae Suzy,Lee Jong Suk,Jung Hae In', 'Son Ye Jin,Jung Hae In,Jang So Yeon', 'Kim Woo Bin,Bae Suzy,Im Joo Hwan']} Title Cast 0 Crash Landing on You Hyun Bin,Son Ye Jin,Seo Ji Hye 1 Memories of the Alhambra Hyun Bin,Park Shin Hye,Park Hoon 2 The Heirs Lee Min Ho,Park Shin Hye,Kim Woo Bin 3 While You Were Sleeping Bae Suzy,Lee Jong Suk,Jung Hae In 4 Something in the Rain Son Ye Jin,Jung Hae In,Jang So Yeon 5 Uncontrollably Fond Kim Woo Bin,Bae Suzy,Im Joo Hwan  When I split the text and do value counts: df['Cast'] = df['Cast'].str.split(',') df['Cast'].value_counts() [Hyun Bin, Son Ye Jin, Seo Ji Hye] 1 [Hyun Bin, Park Shin Hye, Park Hoon] 1 [Lee Min Ho, Park Shin Hye, Kim Woo Bin] 1 [Bae Suzy, Lee Jong Suk, Jung Hae In] 1 [Son Ye Jin, Jung Hae In, Jang So Yeon] 1 [Kim Woo Bin, Bae Suzy, Im Joo Hwan] 1 Name: Cast, dtype: int64  How do I get the amount of times a specific text is shown in the 'Cast' column? ie: [Park Shin Hye] 2 [Hyun Bin] 2 [Bae Suzy] 1 etc 

Answer: You should use the .explode method to "unpack" each list in different rows. Then .value_counts will work as intended in the original code: import pandas as pd d = {'Title': ['Crash Landing on You', 'Memories of the Alhambra', 'The Heirs', 'While You Were Sleeping', 'Something in the Rain', 'Uncontrollably Fond'], 'Cast' : ['Hyun Bin,Son Ye Jin,Seo Ji Hye', 'Hyun Bin,Park Shin Hye,Park Hoon', 'Lee Min Ho,Park Shin Hye,Kim Woo Bin', 'Bae Suzy,Lee Jong Suk,Jung Hae In', 'Son Ye Jin,Jung Hae In,Jang So Yeon', 'Kim Woo Bin,Bae Suzy,Im Joo Hwan']} df = pd.DataFrame(d) df['Cast'].str.split(',').explode('Cast').value_counts()
Question: I think there should be a question like this already, but I haven't found it. It could be because I don't know the exact concepts/words about what I'm looking for, but here is the example: I have this code: group_1 = ['Hello', 'world', '!'] group_2 = [1,23,4,2,5,2] group_3 = ['A', 'K', 'L'] all_groups = [group_1, group_2, group_3] for i in all_groups: print(i, ':', len(i))  It gives this output: ['Hello', 'world', '!'] : 3 [1, 23, 4, 2, 5, 2] : 6 ['A', 'K', 'L'] : 3  And this is the expected output: 'group_1' : 3 'group_2' : 6 'group_3' : 3  As you can see, I'm trying to print the names of the callable objects group_1, group_2, and group_3. Any suggestions?

Answer: Restructure your code so that it uses a dictionary to store the group names. I would not recommend approaches that use anything related to reflection, the inspect module, or locals(), as described (or linked to) in the comments. The names of the variables in all_groups list aren't preserved when you add them to all_groups; even if they were, accessing these names would likely be more complex than just using a dictionary: data = { 'group_1': ['Hello', 'world', '!'], 'group_2': [1,23,4,2,5,2], 'group_3': ['A', 'K', 'L'] } for k, v in data.items(): print(k, ':', len(v))  This outputs: group_1 : 3 group_2 : 6 group_3 : 3
Question: In trying to understand why my maximum batch size is limited for my PyTorch model, I noticed that it's not the model itself nor loading the tensors onto the GPU that uses the most memory. Most memory is used up when generating a prediction for the first time, e.g. with the following line in the training loop: output = model(images)  where images is some input tensor, and model is my PyTorch model. Before running the line, I have something like 9GB of GPU memory available, and afterwards I'm down to 2.5GB (it then further drops to 1GB available after running loss = criterion(outputs, labels). Two questions:  Is this normal? Why is it happening? What is all that memory being used for? From what I understand the model is already loaded in, and the actual input tensors are already on the GPU before making that call. The output tensors themselves can't be that big. Does it have something to do with storing the computational graph? 

Answer: This is normal: The key here is that all intermediate tensors (the whole computation graph) have to be stored if you want to compute the gradient via backward-mode differentiation. You can aviod that by using the .no_grad context manager: with torch.no_grad(): output = model(images)  You will observe that a lot less memory is used, because no computation graph will be stored. But this also means that you can't compute the derivatives anymore. It is however the standard way if you just want to evaluate the model without the need of any optimization. There is one way to reduce the memory cosumption if you still want to optimize, and it is called checkpointing. Whenever you need an intermediate tensor in the backward pass, it will be computed again from the input (or actually from the last "checkpoint"), without storing an intermediate tensor up to that tensor. But this is just computationally more expensive. You're trading memory against computational time.
Question: We are trying to automate process using selenium python for a website but as we proceed with the process the bot gets detected every time and a captcha comes up. Even though a human solve that captcha the website does not allow to move forward and continuously keeps detecting the bot and again and again shows the captcha to solve. So far we have tried all methods whichever we have explored to overcome it but none of method worked. Can someone help on this issue ? Some of the methods tried: 1.) using spoofing user agents 2.) using proxies 3.) turn-off useAutomationExtension 4.) changing the property value of navigator for web driver to undefined 5.) disable-blink-features 6.) exclude the collection of enable-automation switches

Answer: There is something called "Undetected ChromeDriver" you can check out! Optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io Automatically downloads the driver binary and patches it. here is the link Here is another useful website you can check out, this website shows if a site will detect you using selenium or not or anything like that: LINK Also for future reference on Stack Overflow you should steer away from opinion-based questions. Read this to learn more about asking a good question.
Question: I want this code to append 1 for the first occurence of an item in list p that occurs in list s, and append 0 for the other occurence and other items in s. That's my current code below and it is appending 1 for all occurences, I want it to append 1 for the first occurence alone. Please, help s = [20, 39, 0, 87, 13, 0, 23, 56, 12, 13] p = [0, 13] bin = [] for i in s: if i in p: bin.append(1) else: bin.append(0) print(bin) # current result [0, 0, 1, 0, 1, 1, 0, 0, 0, 1] # excepted result [0, 0, 1, 0, 1, 0, 0, 0, 0, 0] 

Answer: The simplest solution is to remove the item from list p if found: s = [20, 39, 0, 87, 13, 0, 23, 56, 12, 13] p = [0, 13] out = [] for i in s: if i in p: out.append(1) p.remove(i) else: out.append(0) print(out)  Prints: [0, 0, 1, 0, 1, 0, 0, 0, 0, 0]
Question: I was wondering if there is a quick way to normalize the KDE curves (such that the integral of each curve is equal to one) for two displayed sample batches (see figure below). So far I use: sb.displot(data=proc, x="TPSA", hue="Data", kind="kde", legend=False)  Giving me the following plot: non-normalized KDE Plot. Thanks in advance for the help.

Answer: When the hue parameter is set, seaborn by default normalises with respect to the area of all kde curves combined. If you'd like to normalise each curve independently (so the area under each curve is 1), you should provide the displot/kdeplot with common_norm=False. e.g. in your case sb.displot(data=proc, x="TPSA", hue="Data", kind="kde", legend=False, common_norm=False)
Question: As far as I know, python list is a dynamic array. So when we reach a certain size, the capacity of that list will be increased automatically. But the problem is, unlike dynamic array of c or c++, even after increasing the capacity of list instance, the location is not being changed. Why is it happening? I've tested this using the following code block l = [] print(l.__sizeof__()) print(id(l)) for i in range(5_000_000): l.append(i) print(l.__sizeof__()) print(id(l)) 

Answer: In CPython (the implementation written in C distributed by python.org), a Python object never moves in memory. In the case of a list object, two pieces of memory are actually allocated: a basic header struct common to all variable-size Python container objects (containing things like the reference count, a pointer to the type object, and the number of contained objects), and a distinct block of memory for a C-level vector holding pointers to the contained Python objects. The header struct points to that vector. That vector can change size in arbitrary ways, and the header struct will change to point to its current location, but the header struct never moves. id() returns the address of that header struct. Python does not expose the address of the vector of objects.
Question: I have an Enum where I would like a default member to be returned when a member does not exist inside of it. For example: class MyEnum(enum.Enum): A = 12 B = 24 CUSTOM = 1 print(MyEnum.UNKNOWN) # Should print MyEnum.CUSTOM  I know I can use a metaclass like so: class MyMeta(enum.EnumMeta): def __getitem__(cls, name): try: return super().__getitem__(name) except KeyError as error: return cls.CUSTOM class MyEnum(enum.Enum,metaclass=MyMeta): ...  But that appears to only work if I access the Enum using MyEnum['UNKNOWN']. Is there a way that covers both methods of accessing members of an enum when the member doesn't exist?

Answer: Add a definition for __getattr__ to the metaclass: class MyMeta(enum.EnumMeta): def __getitem__(cls, name): try: return super().__getitem__(name) except KeyError as error: return cls.CUSTOM def __getattr__(cls, name): try: return super().__getattr__(name) except AttributeError as error: return cls.CUSTOM  Then, your code will output: MyEnum.CUSTOM
Question: I am wondering, if python GIL allow only a single thread / process to run at once, why should I use asyncio, I get that switching between threads is expensive but, thats it? this is the only advantage of asyncio in python?

Answer: Threading in Python is inefficient because of the GIL (Global Interpreter Lock) which means that multiple threads cannot be run in parallel as you would expect on a multi-processor system. Plus you have to rely on the interpreter to switch between threads, this adds to the inefficiency. asyc/asyncio allows concurrency within a single thread. This gives you, as the developer, much more fine grained control of the task switching and can give much better performance for concurrent I/O bound tasks than Python threading. The 3rd approach that you don't mention is multiprocessing. This approach uses processes for concurrency and allows programs to make full use of hardware with multiple cores.
Question: I hava a list of 10,000 random sets with different lengths: import random random.seed(99) lst = [set(random.sample(range(1, 10000), random.randint(1, 1000))) for _ in range(10000)]  I want to know the fastest way to check if there is any set that is a subset of another set (or equivalently if there is any set that is a superset of another set). Right now I am using the following very basic code: def any_containment(lst): checked_sets = [] for st in lst: if any(st.issubset(s) for s in checked_sets): return True else: checked_sets.append(st) return False %timeit any_containment(lst) # 12.3 ms ± 230 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  Clearly, my code is not utilizing previous information when checking containment in each iteration. Can anyone suggest the fastest way to do this?

Answer: Seems to be faster to sort by length and then try small sets as subset first (and for each, try large sets as superset first). Times in ms from ten cases, data generated like you did but without seeding: agree yours mine ratio result True 2.24 2.98 0.75 True True 146.25 3.10 47.19 True True 121.66 2.90 41.91 True True 0.21 2.73 0.08 True True 37.01 2.82 13.10 True True 5.86 3.13 1.87 True True 54.61 3.14 17.40 True True 0.86 2.81 0.30 True True 182.51 3.06 59.60 True True 192.93 2.73 70.65 True  Code (Try it online!): import random from timeit import default_timer as time def original(lst): checked_sets = [] for st in lst: if any(st.issubset(s) for s in checked_sets): return True else: checked_sets.append(st) return False def any_containment(lst): remaining = sorted(lst, key=len, reverse=True) while remaining: s = remaining.pop() if any(s <= t for t in remaining): return True return False for _ in range(10): lst = [set(random.sample(range(1, 10000), random.randint(1, 1000))) for _ in range(10000)] t0 = time() expect = original(lst) t1 = time() result = any_containment(lst) t2 = time() te = t1 - t0 tr = t2 - t1 print(result == expect, '%6.2f ' * 3 % (te*1e3, tr*1e3, te/tr), expect)  Improvement The following seems further ~20% faster. Instead of first comparing the smallest set with potentially all larger sets before giving even just the second-smallest a chance, this does give other small sets an early chance. def any_containment(lst): sets = sorted(lst, key=len) for i in range(1, len(sets)): for s, t in zip(sets, sets[-i:]): if s <= t: return True return False  Comparison with my old solution (Try it online!): agree old new ratio result True 3.13 2.46 1.27 True True 3.36 3.31 1.02 True True 3.10 2.49 1.24 True True 2.72 2.43 1.12 True True 2.86 2.35 1.21 True True 2.65 2.47 1.07 True True 5.24 4.29 1.22 True True 3.01 2.35 1.28 True True 2.72 2.28 1.19 True True 2.80 2.45 1.14 True  Yet another idea A shortcut could be to first collect the union of all single-element sets, and check whether that intersects with any other set (either without sorting them, or again from largest to smallest after sorting). That likely suffices. If not, then proceed as previously, but without the single-element sets.
Question: We currently have a python script launched locally that periodically generates dozens of Excel files using Xlwings. How can it be deployed on a cloud server as an ETL that would be linked to a job scheduler, so that no human action is needed anymore? My concern is that Xlwings requires an Excel license (and a GUI?), which is not usually available in the production server.

Answer: The only way that you can currently do what you have in mind is to install Excel, Python, and xlwings on a Windows Server: xlwings was built for interactive workflows. You might want to look into OpenPyXL and XlsxWriter to see if you can create the reports by writing the Excel file directly, as opposed to automating the Excel application, as xlwings does.
Question: I am trying to interpret the value that I get out of sklearn.metrics.mean_absolute_percentage_error(y_true, y_pred), but have difficulty to understand the interpretation. I need to interpret the result based on below accepted (?) schema  Based on the official Python explanation: Note here that the output is not a percentage in the range [0, 100] and a value of 100 does not mean 100% but 1e2. Furthermore, the output can be arbitrarily high when y_true is small (which is specific to the metric) or when abs(y_true - y_pred) is large (which is common for most regression metrics). Read more in the User Guide. So again, from sklearn.metrics import mean_absolute_percentage_error y_true = [3, -0.5, 2, 7] y_pred = [2.5, 0.0, 2, 8] mean_absolute_percentage_error(y_true, y_pred) 0.3273...  what does 0.32 mean? if this does not mean 32%, then what? I am using this function and I am getting these results for two different data sets: 0.3 for one set of my data 1.3 for another set of my data I can say the first set is more accurate, but can I say 30% is MAPE of the first set and 130% is MAPE of the second set, I guess I cannot. So How do I need to interpret these outputs?

Answer: If you look at the source code for the mape calculation in sklearn you will see the value is not multiplied by 100, so it is not a percentage. Therefore, while interpreting your results, you should multiply the mape value by a 100 to have it in percentage. You must also pay a close attention to your actual data if there is value close to 0 then they could cause mape to be large. For instance, you could look at the wikipedia link on mape formulation. I suggest you plot the actual value and the predicted value as a scatter plot and then compare it with the line y=x.