eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I got gensim to work in Google Collab by following this process: !pip install gensim from gensim.summarization import summarize  Then I was able to call summarize(some_text) Now I'm trying to run the same thing in VS code: I've installed gensim: pip3 install gensim but when I run from gensim.summarization import summarize  I get the error Import "gensim.summarization" could not be resolvedPylancereportMissingImports  I've also tried from gensim.summarization.summarizer import summarize with same error. Regardless I haven't been able to call the function summarize(some_text) outside of Google Collab.

Answer: The summarization code was removed from Gensim 4.0. See: https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4#12-removed-gensimsummarization  12. Removed gensim.summarization Despite its general-sounding name, the module will not satisfy the majority of use cases in production and is likely to waste people's time. See this Github ticket for more motivation behind this.  If you need it, you could try:  installing an older gensim version (such as 3.8.3, the last official release in which it remained); or… copy the source code out to your own local module  However, I expect you'd likely be disappointed by its inflexibility and how little it can do. It was only extractive summarization - choosing a few key sentences from those that already exist. That only gives impressive results when the source text was already well-written in an expository style mixing high-level overview sentences with separate detail sentences. And, its method of analyzing/ranking words was very crude & hard-to-customize – totally unconnected to the more generic/configurable/swappable approaches used elsewhere in Gensim or in other text libraries.
Question: I am looking for a fast method to traverse a list of sets, and to expand each set by finding its union with any other element of the list with which it shares at least one element. For example, suppose that I have four rows of data, where each row corresponds to a set of unique elements 0, 5, 101 8, 9, 19, 21 78, 79 5, 7, 63, 64  The first and the last rows have the intersecting element 5 and so after performing my operation I want to have the unions 0, 5, 7, 63, 64, 101 8, 9, 19, 21 78, 79 0, 5, 7, 63, 64, 101  Right now, I can nearly do this with two loops: def consolidate_list(arr): """ arr (list) : A list of lists, where the inner lists correspond to sets of unique integers """ arr_out = list() for item1 in arr: item_additional = list() # a list containing all overlapping elements for item2 in arr: if len(np.intersect1d(item1, item2)) > 0: item_additional.append(np.copy(item2)) out_val = np.unique(np.hstack([np.copy(item1)] + item_additional)) # find union of all lists arr_out.append(out_val) return arr_out  The issue with this approach is that it needs to be run multiple times, until the output stops changing. Since the input might be jagged (ie, different numbers of elements per set), I can't see a way to vectorize this function.

Answer: This problem is about creating disjoint sets and so I would use union-find methods. Now Python is not particularly known for being fast, but for the sake of showing the algorithm, here is an implementation of a DisjointSet class without libraries: class DisjointSet: class Element: def __init__(self): self.parent = self self.rank = 0 def __init__(self): self.elements = {} def find(self, key): el = self.elements.get(key, None) if not el: el = self.Element() self.elements[key] = el else: # Path splitting algorithm while el.parent != el: el, el.parent = el.parent, el.parent.parent return el def union(self, key=None, *otherkeys): if key is not None: root = self.find(key) for otherkey in otherkeys: el = self.find(otherkey) if el != root: # Union by rank if root.rank < el.rank: root, el = el, root el.parent = root if root.rank == el.rank: root.rank += 1 def groups(self): result = { el: [] for el in self.elements.values() if el.parent == el } for key in self.elements: result[self.find(key)].append(key) return result  Here is how you could use it for this particular problem: def solve(lists): disjoint = DisjointSet() for lst in lists: disjoint.union(*lst) groups = disjoint.groups() return [lst and groups[disjoint.find(lst[0])] for lst in lists]  Example call: data = [ [0, 5, 101], [8, 9, 19, 21], [], [78, 79], [5, 7, 63, 64] ] result = solve(data)  The result will be: [[0, 5, 101, 7, 63, 64], [8, 9, 19, 21], [], [78, 79], [0, 5, 101, 7, 63, 64]]  Note that I added an empty list in the input list, so to illustrate that this boundary case remains unaltered. NB: There are libraries out there that provide union-find/disjoint set functionality, each with a slightly different API, but I suppose that using one of those can give a better performance.
Question: I want to check if the type of a variable is: dict[str, Any]. (in python) What I have tried (unsuccessfully) is this: myvar = { 'att1' : 'some value', 'att2' : 1 } if not isinstance(myvar, dict[str, Any]): raise Exception('Input has the wrong type')  I get the following error message:  TypeError: isinstance() argument 2 cannot be a parameterized generic  How should I do this? Thank you!

Answer: Try the below - make sure you have a dict and the keys of the dict are strings. data1 = { 'att1': 'some value', 'att2': 1 } data2 = { 'att1': 'some value', 13: 1 } def check_if_dict_with_str_keys(data): return isinstance(data, dict) and all(isinstance(x, str) for x in data.keys()) print(check_if_dict_with_str_keys(data1)) print(check_if_dict_with_str_keys(data2))  output True False
Question: I have a list of 1 and 2, e.g. [2, 1, 1, 1] I need to get all possible combinations: [[2, 1, 1, 1], [1, 2, 1, 1], [1, 1, 2, 1], [1, 1, 1, 2]]  I tried to use itertools' product, however, it return the same result (e.g. [2, 1, 1, 1]) multiple times, and it is inefficient when input is bigger. Is there some build in function for something like this?

Answer: What you are looking for is permutations: >>> import itertools >>> a = [2, 1, 1, 1] >>> list(set(itertools.permutations(a))) [(1, 1, 1, 2), (1, 1, 2, 1), (2, 1, 1, 1), (1, 2, 1, 1)]
Question: I have three lists: list_01 = ['DOG','CAT','BEAR'] list_02 = ['V','W','X','Y','Z'] list_03 = ['A','B','C','D','E','F','G','H']  What I hope to get is a list like the following: list_04 = ['DOG','V','A','CAT','W','B','BEAR','X','C','Y','D','Z','E','F','G','H']  This list is supposed to contain one item from list 1, then one from list 2, and one from list 3. This then continues until list 1 is exhausted; list 1 should then be ignored, and the same process should happen on just lists 2 and 3, continuing until all lists are empty.

Answer: It seems like you want to do this in order, not randomly. If so, you can use zip_longest() from itertools and make a nested list comprehension: from itertools import zip_longest list_01 = ['DOG','CAT','BEAR'] list_02 = ['V','W','X','Y','Z'] list_03 = ['A','B','C','D','E','F','G','H'] list_04 = [n for group in zip_longest(list_01, list_02, list_03) for n in group if n is not None] # ['DOG', 'V', 'A', 'CAT', 'W', 'B', 'BEAR', 'X', 'C', 'Y', 'D', 'Z', 'E', 'F', 'G', 'H']  Note: zip_longest will produce None values when one list runs out. That's why we are filtering for None in the comprehension.
Question: How could I format the bar labels to remove the scientific notation? highest_enrollment = course_data.groupby( "course_organization")["course_students_enrolled"].sum().nlargest(10) ax = sns.barplot(x=highest_enrollment.index, y=highest_enrollment.values, ci=None, palette="ch: s=.5, r=-.5") ax.ticklabel_format(style='plain', axis="y") plt.xticks(rotation=90) ax.bar_label(ax.containers[0]) plt.show()  

Answer: As already suggested by BigBen in the comment, you can pass fmt parameter to matplotlib.axes.Axes.bar_label; you can use %d for integers: import matplotlib.pyplot as plt import seaborn as sns import pandas as pd highest_enrollment = pd.DataFrame({'class': ['A', 'B', 'C'], 'values': [30000000, 20000000, 10000000]}) ax = sns.barplot(data = highest_enrollment, x = 'class', y = 'values', palette="ch: s=.5, r=-.5") ax.ticklabel_format(style='plain', axis="y") plt.xticks(rotation=90) ax.bar_label(ax.containers[0], fmt = '%d') plt.show()
Question: I'm given a list of lists s: s = [["a1", "A"], ["b4", "B"], ["a3", "A"], ["d6", "D"], ["c4", "C"]]  (note that the elements in a list do not necessarily begin with a same letter. I modified the data here for convenience.) My goal is to sort each list to a category by its second element, and get all possible combinations by picking at most one element in each category. I first hashed the list of lists to a dictionary: dic = {i[1]: [] for i in s} for i in s: # set the value of the first item key to the second item dic[i[1]].append(i[0]) dic >>> {'A': ['a1', 'a3'], 'B': ['b4'], 'C': ['c4'], 'D': ['d6']}  The number of all possible combinations, hence the length of a powerset of s, should return 23: {'a1'}, {'a3'}, {'b4'}, {'c4'}, {'d6'}, {'a1', 'b4'}, {'a1', 'c4'}, {'a1', 'd6'}, {'a3', 'b4'}, {'a3', 'c4'}, {'a3', 'd6'}, {'b4', 'c4'}, {'b4', 'd6'}, {'c4', 'd6'}, {'a1', 'b4', 'c4'}, {'a1', 'b4', 'd6'}, {'a1', 'c4', 'd6'}, {'a3', 'b4', 'c4'}, {'a3', 'b4', 'd6'}, {'a3', 'c4', 'd6'}, {'b4', 'c4', 'd6'}, {'a1', 'b4', 'c4', 'd6'}, {'a3', 'b4', 'c4', 'd6'}  I initially was going to put multiple for loops, but since I have no guarantee in how many key I would have in my s (which would also put my time complexity at O(N^x)), I used itertools.chain and itertools.combinations as per this post: def powerset(s:list): return chain.from_iterable(combinations(s, r) for r in range(1, len(s)+1))  The problem is that this only takes elements in a single list to account, hence neglects the constraint: 'not taking more than one element from each list at most'. Flattening a list would disregard the categories, so I've not attempted to do so. Any insights to tackle this problem would be appreciated.

Answer: @don'ttalkjustcode's answer works but unnecessarily incurs the overhead of adding dummy values, and also produces an empty set, which is not required by the question. A more direct approach would be to use itertools.combinations to pick lists from the dict of lists to pass to itertools.product to produce the desired combinations: from itertools import product, combinations print(*( set(p) for r in range(len(dic)) for c in combinations(dic.values(), r + 1) for p in product(*c) ), sep='\n')  This outputs: {'a1'} {'a3'} {'b4'} {'c4'} {'d6'} {'a1', 'b4'} {'a3', 'b4'} {'a1', 'c4'} {'a3', 'c4'} {'d6', 'a1'} {'d6', 'a3'} {'c4', 'b4'} {'d6', 'b4'} {'d6', 'c4'} {'a1', 'c4', 'b4'} {'a3', 'c4', 'b4'} {'d6', 'a1', 'b4'} {'d6', 'a3', 'b4'} {'d6', 'a1', 'c4'} {'d6', 'a3', 'c4'} {'d6', 'c4', 'b4'} {'d6', 'a1', 'c4', 'b4'} {'d6', 'a3', 'c4', 'b4'}
Question: I would like to make a table that evaluates whether a user is in a group or not. How can I get my dictionary sorted like in the example I have down below? I would like the index and columns populated automatically by the key and value. d = { 'user1': ['group1', 'group2', 'group3'], 'user2': ['group1', 'group2'], 'user3': ['group2']} df = pd.DataFrame.from_dict(d,orient='index') print(df)  Current result  user1 user2 user3 0 group1 group1 group2 1 group2 group2 NaN 2 group3 NaN NaN  Desired result - I would like the rows based on the key and the columns based on the values.  group1 group2 group3 user1 Y Y Y user2 Y Y N user3 N Y N 

Answer: Try: d = { "user1": ["group1", "group2", "group3"], "user2": ["group1", "group2"], "user3": ["group2"], } df = pd.DataFrame.from_dict(d, orient="index") x = df.stack().droplevel(level=1) x = pd.crosstab(x.index, x).replace({1: "Y", 0: "N"}) x.index.name, x.columns.name = None, None print(x)  Prints:  group1 group2 group3 user1 Y Y Y user2 Y Y N user3 N Y N
Question: I am converting a matlab code to Python. In matlab there is a line which converting the complex number to the int16: real = int16(real(-3.406578165491512e+04 + 9.054663292273188e+03i)); imag= int16(imag(-3.406578165491512e+04 + 9.054663292273188e+03i)); real= -32768 imag=9055  In python I have tried this: real = np.int16(round(np.real(-3.406578165491512e+04 + 9.054663292273188e+03j))) imag = np.int16(round(np.imag(-3.406578165491512e+04 + 9.054663292273188e+03j))) real= 31470 imag=9055  The results are different (I have had many other values such as (1.815808483565253e+04 + 3.533772674703890e+04j) with different answer!) would you please help me to get the same answer?

Answer: Wolfie gets at the difference, this is about how to solve it. If you're OK with clipping, then you can use iinfo to get the min and max values of an integer type (or hard-code it, if you know you won't be changing it from int16 ever) and then use clip to constrain the float to be within those bounds before casting it. n = -3.406578165491512e+04 ii = np.iinfo(np.int16) print(f"min = {ii.min}") # min = -32768 print(f"max = {ii.max}") # max = 32767 np.int16(np.clip(n, ii.min, ii.max)) # -32768  IMPORTANT NOTE: This is only reliable if the size if your float is larger than the size of the int, because it relies upon being able to represent ii.max exactly as a float. See here for a discussion of when this is not true. Here's an example of that failing n = np.float64(1e100) ii = np.iinfo(np.int64) print(f"max: {ii.max}") # max = 9223372036854775807 clipped = np.clip(n, ii.min, ii.max) print(f"clipped to: {int(clipped)}") # clipped to: 9223372036854775808 print(f"as int: {np.int64(clipped)}") # as int: -9223372036854775808  (This happens because ii.max cannot be represented as a float. Past 9007199254740992, we lose the 1's place of precision and can only specify even integers, so the bounds of the clipping become incorrect.)
Question: Suppose there are n sets of real numbers: S[1], S[2], ..., S[n]. We know two things about these sets:  Each set S[i] has exactly 3 elements.  All elements in each of the sets S[i] are real numbers in the [0, 1] range. (I don't know if this detail can be helpful for the solution, though).   Let's consider a set T of all numbers that can be represented as p[1] * p[2] * p[3] * ... * p[n] where p[i] is an element of S[i]. This set T, obviously, has 3^n elements. My question is, given the sets S[1], S[2], ..., S[n] (1 <= n <= 30) and some 1 <= k <= 10 as input, can we find the k-th largest number in T faster than in O(3^n) time? It's important that I need not only the k-th largest number, but also the corresponding numbers (p[1], p[2], p[3], ... , p[n]) that produce it. Even if the answer is no, I would appreciate any hints on how you would solve this problem approximately, maybe, by using some heuristics? I know about beam search, but maybe you could suggest something else? And even for beam search, it is not really clear how to implement it here the best way. If the exact answer can be obtained algorithmically in less than O(3^n) time, I would greatly appreciate it if you could point out the solution.

Answer: Well, you know that the largest product is the one that uses the largest factor from each set. Furthermore, every other product can be formed by starting with a larger one, and then decreasing the factor chosen in exactly one set. That leads to a simple search:  Put the largest product in a max-first priority queue.  Repeat k times: a. Remove the largest product p from the priority queue b. For each set that has a smaller number than the one selected in p, generate the product formed by decreasing that number to the next lower one in that set. If this selection of factors hasn't been seen before, then add it to the priority queue.   Products will be removed from the queue in decreasing order, so the kth one you take out is the kth largest. Complexity is about N*(k log kN), depending on how you implement things. Note that there may be multiple ways to select the factors that produce the same product. This solution considers those ways to be distinct products, i.e., each way is counted when finding the kth largest. That may or may not be what you want.
Question: dec = 0.1 data = np.array([100,200,300,400,500])  I have a for loop like this y = np.zeros(len(data)) for i in range(len(data)): if i == 0: y[i] = (1.0 - dec) * data[i] else: y[i] = (1.0 - dec) * data[i] + (dec * y[i - 1])  Output y is: array([ 90. , 189. , 288.9 , 388.89 , 488.889])  And now I want to do the above calculation without a loop, so if I break the code and do data[0] = (1.0 - dec) * data[0] data[1:] = (1.0 - dec) * data[1:] + (dec * data[0])  Output data is: array([ 90, 189, 279, 369, 459])  When you compare y and data output first two values are correct because it is getting multiplied with data[0] which makes sense but later on it should continue as the loop does in loop code, so how can we achieve that? Is there a function that can handle this? This is mainly to optimize my code so that it runs faster for thousands of data. The expected output is the same as the y output.

Answer: We can do this with scipy.linalg.toeplitz to make a matrix of shifts of the data and then multiplying that by powers of dec and summing columns: import numpy as np from scipy.linalg import toeplitz dec = 0.1 data = np.array([100,200,300,400,500]) decs = np.power(dec, np.arange(len(data))) r = np.zeros_like(data) r[0] = data[0] toep = toeplitz(r, data) output = (1 - dec) * np.sum(toep * decs.reshape(-1, 1), axis=0)  First decs is a vector of powers of dec: print(decs) #[1.e+00 1.e-01 1.e-02 1.e-03 1.e-04]  Next, we use toeplitz to make a matrix of shifts of data: print(toep) #[[100 200 300 400 500] # [ 0 100 200 300 400] # [ 0 0 100 200 300] # [ 0 0 0 100 200] # [ 0 0 0 0 100]])  Finally we reshape decs into a column, multiply it by toep and sum along columns. This result needs to be scaled by 1 - dec. This all works because we can rewrite our definition of data[i] as a sum instead of recursively: y[i] = (1.0 - dec) * data[i] + (dec * y[i - 1]) y[i] = (1.0 - dec) * data[i] + (dec * ((1.0 - dec) * data[i - 1] + (dec * y[i - 2]))) ... y[i] = (1.0 - dec) * (data[i] + dec * data[i - 1] + dec ** 2 * data[i - 2] + ... dec ** i * data[0]) y[i] = (1.0 - dec) * sum(dec ** j * data[i - j] for j in range(i + 1))  This can be proven by induction. From there it follows from rewriting those sums as columns of a matrix and translating that matrix to a calculation in numpy/scipy.
Question: I've read through the documentation and this doesn't seem to be working for me. I followed this doc. But I'm not sure if it's related to what I'm trying to do, I think this doc is for passing queries like this - site.com/endpoint?keyword=test Here's my goal: api.site.com/test/(optional_field) So, if someone goes to the /test endpoint then it defaults the optional field to a parameter but if they add something there then it takes that as a input. With that said, here's my code:  @app.get("/company/{company_ticker}/model/{financialColumn}", dependencies=[Depends(api_counter)]) async def myendpoint( company_ticker: str, financialColumn: Optional[str] = 'netincome', .. myFunction(company_ticker, financialColumn)  what I'm trying to do is if they just go to the endpoint without the optional flag then it defaults to 'netincome' but if they add something then financialColumn is set to that value. Is there something I can do?

Answer: As far as I know, it won't work the way you've set it up. Though you can try something like this: @app.get("/company/{company_ticker}/model/", dependencies=[Depends(api_counter)]) @app.get("/company/{company_ticker}/model/{financialColumn}", dependencies=[Depends(api_counter)]) async def myendpoint( company_ticker: str, financialColumn: Optional[str] = 'netincome' ): myFunction(company_ticker, financialColumn)  This way, if someone goes to "/company/{company_ticker}/model/" or "/company/{company_ticker}/model/blabla" the function myendpoint will handle the request. Not sure if it works as you wish, but at the moment I cannot test it. Maybe later. Let me know.
Question: mypy version 0.910 Consider d = { 'a': 'a', 'b': { 'c': 1 } } d['b']['d'] = 'b'  Feeding this to mypy results with error: Unsupported target for indexed assignment ("Collection[str]")  Putting a side that mypy inferred the wrong type for d (it is clearly not a collection of strings), adding a very basic explicit type for d fixes this: d: dict = { ... # same as above }   Success: no issues found in 1 source file  I find this very peculiar. mypy should definitely be able to infer that d is a dict without d: dict.

Answer: d is not being inferred as a collection of strings. It is being inferred as a dict, but dicts take two type variables, one for the keys and one for the values. If we use reveal_type: d = { 'a': 'a', 'b': { 'c': 1 } } reveal_type(d) d['b']['d'] = 'b'  I get: (py39) jarrivillaga-mbp16-2019:~ jarrivillaga$ mypy --version mypy 0.910 (py39) jarrivillaga-mbp16-2019:~ jarrivillaga$ mypy scratch.py scratch.py:7: note: Revealed type is "builtins.dict[builtins.str*, typing.Collection*[builtins.str]]" scratch.py:8: error: Unsupported target for indexed assignment ("Collection[str]") Found 1 error in 1 file (checked 1 source file)  So, it is being inferred as: builtins.dict[builtins.str*, typing.Collection*[builtins.str]] which is a dict mapping strings to collections of strings. This is because for your dict, you have used str and dict[str, int] as values, and I can only surmise that typing.Collection[str] is the least broad type that encompasses str and dict[str, str] ha both. I'm not really sure how mypy is supposed to handle the inference of nested dict literals like yours. Note, annotating with just dict means it is going to use dict[Any, Any], which you probably don't want. You should try to give a more constrained type, but that depends on how you intend to use d.
Question: I am trying to create a plot containing 8 subplots (4 rows and 2 columns). To do so, I have made this code that reads the x and y data and plots it in the following fashion: fig, axs = plt.subplots(4, 2, figsize=(15,25)) y_labels = ['k0', 'k1'] for x in range(4): for y in range(2): axs[x, y].scatter([i[x] for i in X_vals], [i[y] for i in y_vals]) axs[x, y].set_xlabel('Loss') axs[x, y].set_ylabel(y_labels[y])  This gives me the following result:  However, I want to add a title to all the rows (not the plots) in the following way(the titles in yellow text):  I found this image and some ways to do that here but I wasn't able to implement this for my use case and got an error. This is what I tried : gridspec = axs[0].get_subplotspec().get_gridspec() subfigs = [fig.add_subfigure(gs) for gs in gridspec] for row, subfig in enumerate(subfigs): subfig.suptitle(f'Subplot row title {row}')  which gave me the error : 'numpy.ndarray' object has no attribute 'get_subplotspec' So I changed the code to : gridspec = axs[0, 0].get_subplotspec().get_gridspec() subfigs = [fig.add_subfigure(gs) for gs in gridspec] for row, subfig in enumerate(subfigs): subfig.suptitle(f'Subplot row title {row}')  but this returned the error : 'Figure' object has no attribute 'add_subfigure'

Answer: The solution in the answer that you linked is the correct one, however it is specific for the 3x3 case as shown there. The following code should be a more general solution for different numbers of subplots. This should work provided your data and y_label arrays/lists are all the correct size. Note that this requires matplotlib 3.4.0 and above to work: import numpy as np import matplotlib.pyplot as plt # random data. Make sure these are the correct size if changing number of subplots x_vals = np.random.rand(4, 10) y_vals = np.random.rand(2, 10) y_labels = ['k0', 'k1'] # change rows/cols accordingly rows = 4 cols = 2 fig = plt.figure(figsize=(15,25), constrained_layout=True) fig.suptitle('Figure title') # create rows x 1 subfigs subfigs = fig.subfigures(nrows=rows, ncols=1) for row, subfig in enumerate(subfigs): subfig.suptitle(f'Subplot row title {row}') # create 1 x cols subplots per subfig axs = subfig.subplots(nrows=1, ncols=cols) for col, ax in enumerate(axs): ax.scatter(x_vals[row], y_vals[col]) ax.set_title("Subplot ax title") ax.set_xlabel('Loss') ax.set_ylabel(y_labels[col])  Which gives: