eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a dict in python that follows this general format: {'field': ['$.name'], 'group': 'name', 'function': 'some_function'}  I want to do some pre-check of the dict to ensure that 'field' always exists, and that no more keys exist beyond 'group' and 'function' which are both optional. I know I can do this by using a long and untidy if statement, but I'm thinking there must be a cleaner way? This is what I currently have: if (('field' in dict_name and len(dict_name.keys()) == 1) or ('group' in dict_name and len(dict_name.keys()) == 2) or ('function' in dict_name and len(dict_name.keys()) == 2) or ('group' in dict_name and 'function' in dict_name and len(dict_name.keys()) == 3))  Essentially I'm first checking if 'field' exists as this is required. I'm then checking to see if it is the only key (which is fine) or if it is a key alongside 'group' and no others, or a key alongside 'function' and no others or a key alongside both 'group' and 'function' and no others. Is there a tidier way of checking the keys supplied are only these 3 keys where two are optional?

Answer: As far as I'm concerned you want to check, that  The set {'field'} is always contained in the set of your dict keys The set of your dict keys is always contained in the set {'field', 'group', 'function'}  So just code it! required_fields = {'field'} allowed_fields = required_fields | {'group', 'function'} d = {'field': 123} # Set any value here if required_fields <= d.keys() <= allowed_fields: print("Yes!") else: print("No!")  This solution is scalable for any sets of required and allowed fields unless you have some special conditions (for example, mutually exclusive keys) (thanks to @Duncan for a very elegant code reduction)
Question: I tried to execute some project. But I've got an attribute error. I checked my Tensorflow and Keras version. Name: tensorflow Version: 2.3.1 Name: Keras Version: 2.4.3 Summary: Deep Learning for humans python 3.8.2  The code is here. self.dim_ordering = K.common.image_dim_ordering()  Error message: self.dim_ordering = K.common.image_dim_ordering() AttributeError: module 'keras.backend' has no attribute 'common'  Is it okay to use K.image_data_format() instead of k.common.image_dim_ordering() ?

Answer: Yes. It is okay to use k.image_data_format() In Keras v2 the method has been renamed to image_data_format
Question: I am having this error TypeError: 'StudentSubjectGrade' object is not subscriptable of course the data filtered is exist in the database, and i am sure that the filter is correct. what should i do to correct this ? note: this is recycle question, please dont mind the comment below, def SummaryPeriod(request): period = request.GET.get('period') subject = request.GET.get('subject') teacher = request.GET.get('teacher') print(period, "period", "subject", subject) cate = gradingCategories.objects.all() students = StudentSubjectGrade.objects.filter( grading_Period=period).filter( Subjects=subject).filter( Teacher = teacher ) print(students) Categories = list(cate.values_list('id', flat=True).order_by('id')) table = [] student_name = None table_row = None columns = len(Categories) + 1 table_header = ['Student Names'] table_header.extend(list(cate.values('CategoryName', 'PercentageWeight'))) table.append(table_header) for student in students: if not student['Students_Enrollment_Records__Students_Enrollment_Records__Student_Users__Lastname'] + ' ' + \ student[ 'Students_Enrollment_Records__Students_Enrollment_Records__Student_Users__Firstname'] == student_name: if not table_row is None: table.append(table_row) table_row = [None for d in range(columns)] student_name = student[ 'Students_Enrollment_Records__Students_Enrollment_Records__Student_Users__Lastname'] + ' ' + \ student['Students_Enrollment_Records__Students_Enrollment_Records__Student_Users__Firstname'] table_row[0] = student_name id = student['id'] table_row.append(id) table_row[Categories.index(student['Grading_Categories']) + 1] = student['Average'] * student[ 'Grading_Categories__PercentageWeight'] / 100 table.append(table_row) return render(request, 'Homepage/summaryPeriod.html', {'table': table, "teacher": teacher, "subject": subject, "period": period})  this is my traceback Internal Server Error: /SummaryPeriod/ Traceback (most recent call last): File "C:\Users\USER\Desktop\venv\lib\site-packages\django\core\handlers\exception.py", line 47, in inner response = get_response(request) File "C:\Users\USER\Desktop\venv\lib\site-packages\django\core\handlers\base.py", line 179, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "C:\Users\Desktop\Homepage\views.py", line 2693, in SummaryPeriod if not student['Students_Enrollment_Records__Students_Enrollment_Records__Student_Users__Lastname'] + ' ' + \ TypeError: 'StudentSubjectGrade' object is not subscriptable [01/Dec/2020 21:21:01] "GET /SummaryPeriod/?period=3&subject=18&teacher=5 HTTP/1.1" 500 70398 

Answer: TypeError: 'StudentSubjectGrade' object is not subscriptable  this means that student is not a dictionary, you cannot use student['key'] to get what you want. you should use student.sth instead.
Question: Following "Dependabot is moving natively into GitHub!", I had to update my dependabot config files to use version 2 format. My .dependabot/config.yaml did look like: version: 1 update_configs: - package_manager: "python" directory: "/" update_schedule: "live" automerged_updates: - match: dependency_type: "all" update_type: "all"  I've got the following working: version: 2 updates: - package-ecosystem: pip directory: "/" schedule: interval: daily  but I can't seem to add the automerge option again (when checking with the dependabot validator)?

Answer: Here is one solution that doesn't require any additional marketplace installations (originally found here). Simply create a new GitHub workflow (e.g. .github/workflows/dependabotautomerge.yml) containing: name: "Dependabot Automerge - Action" on: pull_request: jobs: worker: runs-on: ubuntu-latest if: github.actor == 'dependabot[bot]' steps: - name: automerge uses: actions/github-script@0.2.0 with: script: | github.pullRequests.createReview({ owner: context.payload.repository.owner.login, repo: context.payload.repository.name, pull_number: context.payload.pull_request.number, event: 'APPROVE' }) github.pullRequests.merge({ owner: context.payload.repository.owner.login, repo: context.payload.repository.name, pull_number: context.payload.pull_request.number }) github-token: ${{github.token}}  There are also various third-party solutions available on GitHub Marketplace.
Question: Given some classifier (SVC/Forest/NN/whatever) is it safe to call .predict on the same instance concurrently from different threads? From a distant point of view, my guess is they do not mutate any internal state. But I did not find anything in the docs about it. Here is a minimal example showing what I mean: #!/usr/bin/env python3 import threading from sklearn import datasets from sklearn import svm from sklearn.ensemble import RandomForestClassifier from sklearn.neural_network import MLPClassifier X, y = datasets.load_iris(return_X_y=True) # Some model. Might be any type, e.g.: clf = svm.SVC() clf = RandomForestClassifier(), clf = MLPClassifier(solver='lbfgs') clf.fit(X, y) def use_model_for_predictions(): for _ in range(10000): clf.predict(X[0:1]) # Is this safe? thread_1 = threading.Thread(target=use_model_for_predictions) thread_2 = threading.Thread(target=use_model_for_predictions) thread_1.start() thread_2.start() 

Answer: Check out this Q&A, the predict and predict_proba methods should be thread safe as they only call NumPy, they do not affect model itself in any case so answer to your question is yes. You can find some info as well in replies here. For example in naive bayes the code is following: def predict(self, X): """ Perform classification on an array of test vectors X. Parameters ---------- X : array-like of shape (n_samples, n_features) Returns ------- C : ndarray of shape (n_samples,) Predicted target values for X """ check_is_fitted(self) X = self._check_X(X) jll = self._joint_log_likelihood(X) return self.classes_[np.argmax(jll, axis=1)]  You can see that the first two lines are only checks for input. Abstract method _joint_log_likelihood is the one that interests us, described as: @abstractmethod def _joint_log_likelihood(self, X): """Compute the unnormalized posterior log probability of X I.e. ``log P(c) + log P(x|c)`` for all rows x of X, as an array-like of shape (n_classes, n_samples). Input is passed to _joint_log_likelihood as-is by predict, predict_proba and predict_log_proba. """  And finally for example for multinominal NB the function looks like (source): def _joint_log_likelihood(self, X): """ Compute the unnormalized posterior log probability of X, which is the features' joint log probability (feature log probability times the number of times that word appeared in that document) times the class prior (since we're working in log space, it becomes an addition) """ joint_prob = X * self.feature_log_prob_.T + self.class_log_prior_ return joint_prob  You can see that there is nothing thread unsafe in predict. Of course you can go through codes and check that for any of those classifiers :)
Question: How to get the most frequent row in a DataFrame? For example, if I have the following table:  col_1 col_2 col_3 0 1 1 A 1 1 0 A 2 0 1 A 3 1 1 A 4 1 0 B 5 1 0 C  Expected result:  col_1 col_2 col_3 0 1 1 A  EDIT: I need the most frequent row (as one unit) and not the most frequent column value that can be calculated with the mode() method.

Answer: In Pandas 1.1.0. is possible to use the method value_counts() to count unique rows in DataFrame: df.value_counts()  Output: col_1 col_2 col_3 1 1 A 2 0 C 1 B 1 A 1 0 1 A 1  This method can be used to find the most frequent row: df.value_counts().head(1).index.to_frame(index=False)  Output:  col_1 col_2 col_3 0 1 1 A
Question: I just implemented a Linux command shell in python using only the os library's low level system calls, like fork() and so on. I was wondering how I can implement a key listener that will listen for key (UP|DOWN) to scroll through the history of my shell. I want do do this without using any fancy libraries, but I am also wishing that this is not something super complicated. My code is just about 100 lines of code, so far, and I don't want to create a monster just to get a simple feature :D My thoughts about the problem is, that it should be possible to create a child process with some kind of loop, that will listen for up ^[[A and down ^[[B, key press, and then somehow put the text into my input field, like a normal terminal. So far the thing I am most interested in is the possibility of the key-listener. But next I will probably have to figure out how I will get that text into the input field. About that I am thinking that I probably have to use some of the stdin features that sys provides. I'm only interested in making it work on Linux, and want to continue using low-level system calls, preferably not Python libraries that handle everything for me. This is a learning exercise.

Answer: By default the standard input is buffered and uses canonical mode. This allows you to edit your input. When you press the enter key, the input can be read by Python. If you want a lower level access to the input you can use tty.setraw() on the standard input file descriptor. This allows you to read one character at a time using sys.stdin.read(1). Note that in this case the Python script will be responsible for handling special characters, and you will lose some of the functionality like character echoing and deleting. For more information take a look at termios(3). You can read about escape sequences which are used for up and down keys on Wikipedia. You should be able to replicate the standard shell behavior if you handle everything in one process. You may also want to try using a subprocess (not referring to the module - you can use fork() or popen()). You would parse the unbuffered input in the main process and send it to stdin (which can be buffered) of the subprocess. You will probably need to have some inter-process communication to share history with the main process. Here is an example of the code needed to capture the input this way. Note that it is only doing some basic processing and needs more work in order to fit your use-case. import sys import tty import termios def getchar(): fd = sys.stdin.fileno() attr = termios.tcgetattr(fd) try: tty.setraw(fd) return sys.stdin.read(1) finally: termios.tcsetattr(fd, termios.TCSANOW, attr) EOT = '\x04' # CTRL+D ESC = '\x1b' CSI = '[' line = '' while True: c = getchar() if c == EOT: print('exit') break elif c == ESC: if getchar() == CSI: x = getchar() if x == 'A': print('UP') elif x == 'B': print('DOWN') elif c == '\r': print([line]) line = '' else: line += c
Question: As I'm given to understand due to the search of issues in the Feather Github, as well as questions in stackoverflow such as What are the differences between feather and parquet?, the Feather format was not recommended as long term storage due to Apache Arrow versions being 0.x.x, and considered volatile due to the continuous new releases. My question is, has this situation changed as of the current Apache Arrow's version, 1.0.1? Is Feather considered stable to use as long term storage?

Answer: Feather files (using the v2 -- default -- format version, not the v1 "legacy" version) are stable starting with Apache Arrow 1.0.0.
Question: Which is the command to see the "correct" CUDA Version that pytorch in conda env is seeing? This, is a similar question, but doesn't get me far.  nvidia-smi says I have cuda version 10.1  conda list tells me cudatoolkit version is 10.2.89  torch.cuda.is_available() shows FALSE, so it sees No CUDA?  print(torch.cuda.current_device()), I get 10.0.10 (10010??) (it looks like):  AssertionError: The NVIDIA driver on your system is too old (found version 10010)   print(torch._C._cuda_getCompiledVersion(), 'cuda compiled version') tells me my version is 10.0.20 (10020??)?  10020 cuda compiled version    Why are there so many different versions? What am I missing? P.S I have Nvidia driver 430 on Ubuntu 16.04 with Geforce 1050. It comes with libcuda1-430 when I installed the driver from additional drivers tab in ubuntu (Software and Updates). I installed pytorch with conda which also installed the cudatoolkit using conda install -c fastai -c pytorch -c anaconda fastai

Answer: In the conda env (myenv) where pytorch is installed do the following: conda activate myenv torch.version.cuda  Nvidia-smi only shows compatible version. Does not seem to talk about the version pytorch's own cuda is built on.
Question: I have such middleware class RequestContext(BaseHTTPMiddleware): async def dispatch(self, request: Request, call_next: RequestResponseEndpoint): request_id = request_ctx.set(str(uuid4())) # generate uuid to request body = await request.body() if body: logger.info(...) # log request with body else: logger.info(...) # log request without body response = await call_next(request) response.headers['X-Request-ID'] = request_ctx.get() logger.info("%s" % (response.status_code)) request_ctx.reset(request_id) return response  So the line body = await request.body() freezes all requests that have body and I have 504 from all of them. How can I safely read the request body in this context? I just want to log request parameters.

Answer: I would not create a Middleware that inherits from BaseHTTPMiddleware since it has some issues, FastAPI gives you a opportunity to create your own routers, in my experience this approach is way better. from fastapi import APIRouter, FastAPI, Request, Response, Body from fastapi.routing import APIRoute from typing import Callable, List from uuid import uuid4 class ContextIncludedRoute(APIRoute): def get_route_handler(self) -> Callable: original_route_handler = super().get_route_handler() async def custom_route_handler(request: Request) -> Response: request_id = str(uuid4()) response: Response = await original_route_handler(request) if await request.body(): print(await request.body()) response.headers["Request-ID"] = request_id return response return custom_route_handler app = FastAPI() router = APIRouter(route_class=ContextIncludedRoute) @router.post("/context") async def non_default_router(bod: List[str] = Body(...)): return bod app.include_router(router)  Works as expected. b'["string"]' INFO: 127.0.0.1:49784 - "POST /context HTTP/1.1" 200 OK
Question: Suppose I have this numpy array: [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]]  My goal is to select two random elements from each row and create a new numpy array that might look something like: [[2, 4], [5, 8], [9, 10], [15, 16]]  I can easily do this using a for loop. However, is there a way that I can use broadcasting, say, with np.random.choice, to avoid having to loop through each row?

Answer: Approach #1 Based on this trick, here's a vectorized way - n = 2 # number of elements to select per row idx = np.random.rand(*a.shape).argsort(1)[:,:n] out = np.take_along_axis(a, idx, axis=1)  Sample run - In [251]: a Out[251]: array([[ 1, 2, 3, 4], [ 5, 6, 7, 8], [ 9, 10, 11, 12], [13, 14, 15, 16]]) In [252]: idx = np.random.rand(*a.shape).argsort(1)[:,:2] In [253]: np.take_along_axis(a, idx, axis=1) Out[253]: array([[ 2, 1], [ 6, 7], [ 9, 11], [16, 15]])  Approach #2 Another based on masks to select exactly two per row - def select_two_per_row(a): m,n = a.shape mask = np.zeros((m,n), dtype=bool) R = np.arange(m) idx1 = np.random.randint(0,n,m) mask[R,idx1] = 1 mask2 = np.zeros(m*(n-1), dtype=bool) idx2 = np.random.randint(0,n-1,m) + np.arange(m)*(n-1) mask2[idx2] = 1 mask[~mask] = mask2 out = a[mask].reshape(-1,2) return out  Approach #3 Another based on integer based indexing again to select exactly two per row - def select_two_per_row_v2(a): m,n = a.shape idx1 = np.random.randint(0,n,m) idx2 = np.random.randint(1,n,m) out = np.take_along_axis(a, np.c_[idx1, idx1 - idx2], axis=1) return out   Timings - In [209]: a = np.random.rand(100000,10) # App1 with argsort In [210]: %%timeit ...: idx = np.random.rand(*a.shape).argsort(1)[:,:2] ...: out = np.take_along_axis(a, idx, axis=1) 23.2 ms ± 137 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) # App1 with argpartition In [221]: %%timeit ...: idx = np.random.rand(*a.shape).argpartition(axis=1,kth=1)[:,:2] ...: out = np.take_along_axis(a, idx, axis=1) 18.3 ms ± 115 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [214]: %timeit select_two_per_row(a) 9.89 ms ± 37.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [215]: %timeit select_two_per_row_v2(a) 5.78 ms ± 9.19 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Question: I have a list of tuples, the list can vary in length between ~8 - 1000 depending on the length of the tuples. Each tuple in the list is unique. A tuple is of length N where each entry is a generic word. An example tuple can be of length N (Word 1, Word 2, Word 3, ..., Word N) For any tuple in the list, element j in said tuple will either be '' or Word j A very simplified example with alphabetic letters would be l = [('A', 'B', '', ''), ('A', 'B', 'C', ''), ('', '', '', 'D'), ('A', '', '', 'D'), ('', 'B', '', '')]  Every position at each tuple will either have the same value or be empty. I want to remove all the tuples which have all their non '' values in another tuple at the same position. As an example, (A,B,'','') has all its non '' values in (A,B,C,'') and should therefore be removed. filtered_l = [(A,B,C,''),(A,'','',D)]  The length of the tuples is always of the same length (not necessarily 4). The length of the tuples would be between 2-10. What is the fastest way to do this?

Answer: Let's conceptualize each tuple as a binary array, where 1 is "contains something" and 2 is "contains an empty string". Since the item at each position will be the same, we don't need to care what is at each position, only that something is. l = [('A','B','',''),('A','B','C',''),('','','','D'),('A','','','D'),('','B','','')] l_bin = [sum(2**i if k else 0 for i,k in enumerate(tup)) for tup in l] # [3, 7, 8, 9, 2] # [0b0011, 0b0111, 0b1000, 0b1001, 0b0010] # that it's backwards doesn't really matter, since it's consistent  Now, we can walk through that list and build a new datastructure without 'duplicates'. Since we have our tuples encoded as binary, we can determine a duplicate, 'encompassed' by another, by doing bitwise operations - given a and b, if a | b == a, then a must contain b. codes = {} for tup, b in zip(l, l_bin): # check if any existing code contains the potential new one # in this case, skip adding the new one if any(a | b == a for a in codes): continue # check if the new code contains a potential existing one or more # in which case, replace the existing code(s) with the new code for a in list(codes): if b | a == b: codes.pop(a) # and finally, add this code to our datastructure codes[b] = tup  Now we can withdraw our 'filtered' list of tuples: output = list(codes.values()) # [('A', 'B', 'C', ''), ('A', '', '', 'D')]  Note that (A, B, C, '') contains both (A, B, '', '') and ('', B, '', ''), and that (A, '', '', D') contains ('', '', '', D), so this should be correct. As of python 3.8, dict preserves insertion order, so the output should be in the same order that the tuples originally appeared in the list. This solution wouldn't be perfectly efficient, since the number of codes might stack up, but it should be between O(n) and O(n^2), depending on the number of unique codes left at the end (and since the length of each tuple is significantly less than the length of l, it should be closer to O(n) than to O(n^2).
Question: Problem I have the following Pandas dataframe:  data = { 'ID': [100, 100, 100, 100, 200, 200, 200, 200, 200, 300, 300, 300, 300, 300], 'value': [False, False, True, False, False, True, True, True, False, False, False, True, True, False], } df = pandas.DataFrame (data, columns = ['ID','value'])  I want to get the following groups:  Group 1: for each ID, all False rows until the first True row of that ID Group 2: for each ID, all False rows after the last True row of that ID Group 3: all true rows   Can this be done with pandas? What I've tried I've tried group = df.groupby((df['value'].shift() != df['value']).cumsum())  but this returns an incorrect result.

Answer: Let us try shift + cumsum create the groupby key: BTW I really like the way you display your expected output s = df.groupby('ID')['value'].apply(lambda x : x.ne(x.shift()).cumsum()) d = {x : y for x ,y in df.groupby(s)} d[2] ID value 2 100 True 5 200 True 6 200 True 7 200 True 11 300 True 12 300 True d[1] ID value 0 100 False 1 100 False 4 200 False 9 300 False 10 300 False d[3] ID value 3 100 False 8 200 False 13 300 False