eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Reading the Django docs, it advices to make a custom creation method for a model named Foo by defining it as create_foo in the manager: class BookManager(models.Manager): def create_book(self, title): book = self.create(title=title) # do something with the book return book class Book(models.Model): title = models.CharField(max_length=100) objects = BookManager() book = Book.objects.create_book("Pride and Prejudice")  My question is that why is the previous one preferred to simply overriding the base class's create method: class BookManager(models.Manager): def create(self, title): book = self.model(title=title) # do something with the book book.save() return book class Book(models.Model): title = models.CharField(max_length=100) objects = BookManager() book = Book.objects.create("Pride and Prejudice")  Imo it seems that only overriding create will prevent anyone from accidentally using it to make a illformed model instance, since create_foo can always be bypassed completely: class BookManager(models.Manager): def create_book(self, title): book = self.create(title=title, should_not_be_set_manually="critical text") return book class Book(models.Model): title = models.CharField(max_length=100) should_not_be_set_manually = models.CharField(max_length=100) objects = BookManager() # Can make an illformed Book!! book = Book.objects.create(title="Some title", should_not_be_set_manually="bad value")  Is there any advantage in doing it like the docs suggest, or is actually overriding create just objectively better?

Answer: Yes, obviously, you can do that. But if you look closer to the example you are quoting from documentation, it is not about whether you should override create or not, it is about  If you do so, however, take care not to change the calling signature as any change may prevent the model instance from being saved.  preserving the calling signature. Because interfaces available for you may also be used by django internally. If you modify them, things may not break for you but for Django. In this example, they are not suggesting this for create but model constructor. Secondly, even standard interface for create is only taking keyword arguments def create(self, **kwargs):  But if you modify it to take positional arguments, def create(self, title): it will break wherever it is used inside Django or in standard way. So you should extend existing functionality not modify and most probably break it.
Question: I can connect to a remote Jupyter Notebook server with a token from VSCode through the "Python: Specify Jupyter server URI" command from the Command Palette.  However, I didn't find a way to do 2 things:  Open an existing Notebook on the remote Jupyter Notebook server. Specify a folder to connect to, where my existing notebook resides in the remote server.  Is there a way of doing it?

Answer: Currently, VSCode doesn't support this functionality. See this issue: https://github.com/microsoft/vscode-python/issues/8161
Question: I have a Luigi task with a boolean parameter that is set to True by default: class MyLuigiTask(luigi.Task): my_bool_param = luigi.BoolParameter(default=True)  When I run this task from terminal, I sometimes want to pass that parameter as False, but get the following result: $ MyLuigiTask --my_bool_param False error: unrecognized arguments: False  Same obviously for false and 0... I understand that I can make the default False and then use the flag --my_bool_param if I want to make it True, but I much prefer to have the default True. Is there any way to do this, and still pass False from terminal?

Answer: Found the solution in Luigi docs: class MyLuigiTask(luigi.Task): my_bool_param = luigi.BoolParameter( default=True, parsing=luigi.BoolParameter.EXPLICIT_PARSING) def run(self): print(self.my_bool_param)  Here EXPLICIT_PARSING tell Luigi that adding the flag --my_bool_param false in the terminal call to MyLuigiTask, will be parsed as store_false. Now we can have: $ MyLuigiTask --my_bool_param false False
Question: I have a Confusion Matrix with really small sized numbers but I can't find a way to change them. from sklearn.metrics import confusion_matrix cm = confusion_matrix(y_test, rf_predictions) ax = plt.subplot() sns.set(font_scale=3.0) #edited as suggested sns.heatmap(cm, annot=True, ax=ax, cmap="Blues", fmt="g"); # annot=True to annotate cells # labels, title and ticks ax.set_xlabel('Predicted labels'); ax.set_ylabel('Observed labels'); ax.set_title('Confusion Matrix'); ax.xaxis.set_ticklabels(['False', 'True']); ax.yaxis.set_ticklabels(['Flase', 'True']); plt.show()  thats the code I am using and the pic I get looks like:  I would not mind changing the numbers of the classification by hand but I dont really want to do it for the labels aswell. EDIT: Figures are bigger now but the labels stay very small Cheers

Answer: Use sns.set to change the font size of the heatmap values. You can specify the font size of the labels and the title as a dictionary in ax.set_xlabel, ax.set_ylabel and ax.set_title, and the font size of the tick labels with ax.tick_params.  from sklearn.metrics import confusion_matrix cm = confusion_matrix(y_test, rf_predictions) ax = plt.subplot() sns.set(font_scale=3.0) # Adjust to fit sns.heatmap(cm, annot=True, ax=ax, cmap="Blues", fmt="g"); # Labels, title and ticks label_font = {'size':'18'} # Adjust to fit ax.set_xlabel('Predicted labels', fontdict=label_font); ax.set_ylabel('Observed labels', fontdict=label_font); title_font = {'size':'21'} # Adjust to fit ax.set_title('Confusion Matrix', fontdict=title_font); ax.tick_params(axis='both', which='major', labelsize=10) # Adjust to fit ax.xaxis.set_ticklabels(['False', 'True']); ax.yaxis.set_ticklabels(['False', 'True']); plt.show()
Question: I recently updated my MacOS to Catalina, and now I have the infamous "zsh command not found: conda" when I enter "conda" in my terminal. I've read a number of solutions, and the easiest for me to try was to reinstall Anaconda in my home directory (specifically, the 2019.10 version of the installer installs in Users/myname/opt/anaconda3), as suggested by the folks at Anaconda here. Well, I did just that and it did not solve the problem. What am I missing?

Answer: From the Anaconda install docs:  In order to initialize after the installation process is done, first run source <path to conda>/bin/activate and then run conda init.  However,  If you are on macOS Catalina, the new default shell is zsh. You will instead need to run source <path to conda>/bin/activate followed by conda init zsh.
Question: Suppose I have a tensor of the form [[-5, 0, -1], [3, 100, 87], [17, -34, 2], [45, 1, 25]]  I want to find the maximum value in each row and return a rank 1 tensor as follows: [0, 100, 17, 45]  How would I do this in PyTorch?

Answer: You can use the torch.max() function. So you can do something like x = torch.Tensor([[-5, 0, -1], [3, 100, 87], [17, -34, 2], [45, 1, 25]]) out, inds = torch.max(x,dim=1)  and this will return the maximum values across each row (dimension 1). It will return max values with their indices.
Question: I am a Python beginner so this may be more obvious than what I'm thinking. I'm using Matplotlib to graphically present my predicted data vs actual data via a neural network. I am able to calculate r-squared, and plot my data, but now I want to combine the value on the graph itself, which changes with every new run. My NN uses at least 4 different inputs, and gives one output. This is my end code for that: y_predicted = model.predict(X_test)  This is how i calculate R2: # Using sklearn from sklearn.metrics import r2_score print r2_score(y_test, y_predicted)  and this is my graph: fig, ax = plt.subplots() ax.scatter(y_test, y_predicted) ax.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=4) ax.set_xlabel('Actual') ax.set_ylabel('Predicted') #regression line y_test, y_predicted = y_test.reshape(-1,1), y_predicted.reshape(-1,1) ax.plot(y_test, LinearRegression().fit(y_test, y_predicted).predict(y_test)) plt.show()  It gives something like the graph attached, and the R2 varies everytime I change the epochs, or number of layers, or type of data etc. The red is my line of regression, which I will label later. Since R2 is a function I can't simply use the legend or text code. I would also like to display MSE. Can anyone help me out? Graph

Answer: If I understand correctly, you want to show R2 in the graph. You can add it to the graph title: ax.set_title('R2: ' + str(r2_score(y_test, y_predicted)))  before plt.show()
Question: I've been experimenting for a while with pd.Series and pd.DataFrame and faced some strange problem. Let's say I have the following pd.DataFrame: df = pd.DataFrame({'col':[[1,2,3]]}) Notice, that this dataframe includes column containing list. I want to modify this dataframe's copy and return its modified version so that the initial one will remain unchanged. For the sake of simplicity, let's say I want to add integer '4' in its cell.  I've tried the following code: def modify(df): dfc = df.copy(deep=True) dfc['col'].iloc[0].append(4) return dfc modify(df) print(df)  The problem is that, besides the new copy dfc, the initial DataFrame df is also modified. Why? What should I do to prevent initial dataframes from modifying? My pandas version is 0.25.0

Answer: From the docs here, in the Notes section:  When deep=True, data is copied but actual Python objects will not be copied recursively, only the reference to the object. This is in contrast to copy.deepcopy in the Standard Library, which recursively copies object data (see examples below).  This is referenced again in this issue on GitHub, where the devs state that:  embedding mutable objects inside a. DataFrame is an antipattern  So this function is working as the devs intend - mutable objects such as lists should not be embedded in DataFrames. I couldn't find a way to get copy.deepcopy to work as intended on a DataFrame, but I did find a fairly awful workaround using pickle: import pandas as pd import pickle df = pd.DataFrame({'col':[[1,2,3]]}) def modify(df): dfc = pickle.loads(pickle.dumps(df)) print(dfc['col'].iloc[0] is df['col'].iloc[0]) #Check if we've succeeded in deepcopying dfc['col'].iloc[0].append(4) print(dfc) return dfc modify(df) print(df)  Output: False col 0 [1, 2, 3, 4] col 0 [1, 2, 3]
Question: I have such DataFrame: df = pd.DataFrame(data={ 'col0': [11, 22,1, 5] 'col1': ['aa:a:aaa', 'a:a', 'a', 'a:aa:a:aaa'], 'col2': ["foo", "foo", "foobar", "bar"], 'col3': [True, False, True, False], 'col4': ['elo', 'foo', 'bar', 'dupa']})  I want to get length of the list after split on ":" in col1, then I want to overwrite the values if length > 2 OR not overwrite the values if length <= 2. Ideally, in one line as fast as possible. Currently, I try but it returns ValueError. df[['col1', 'col2', 'col3']] = df.loc[df['col1'].str.split(":").apply(len) > 2], ("", "", False), df[['col1', 'col2', 'col3']])  EDIT: condition on col1. EDIT2: thank you for all the great and quickly provided answers. amazing! EDIT3: timing on 10^6 rows: @ansev 3.2657s @jezrael 0.8922s @anky_91 1.9511s

Answer: Use Series.str.count, add 1, compare by Series.gt and assign list to filtered columns in list: df.loc[df['col1'].str.count(":").add(1).gt(2), ['col1','col2','col3']] = ["", "", False] print (df) col0 col1 col2 col3 col4 0 11 False elo 1 22 a:a foo False foo 2 1 a foobar True bar 3 5 False dupa
Question: I have a list of dictionaries, and I would like to obtain those that have the same value in a key: my_list_of_dicts = [{ 'id': 3, 'name': 'John' },{ 'id': 5, 'name': 'Peter' },{ 'id': 2, 'name': 'Peter' },{ 'id': 6, 'name': 'Mariah' },{ 'id': 7, 'name': 'John' },{ 'id': 1, 'name': 'Louis' } ]  I want to keep those items that have the same 'name', so, I would like to obtain something like: duplicates: [{ 'id': 3, 'name': 'John' },{ 'id': 5, 'name': 'Peter' },{ 'id': 2, 'name': 'Peter' }, { 'id': 7, 'name': 'John' } ]  I'm trying (not successfully): duplicates = [item for item in my_list_of_dicts if len(my_list_of_dicts.get('name', None)) > 1]  I have clear my problem with this code, but not able to do the right sentence

Answer: Another concise way using collections.Counter: from collections import Counter my_list_of_dicts = [{ 'id': 3, 'name': 'John' },{ 'id': 5, 'name': 'Peter' },{ 'id': 2, 'name': 'Peter' },{ 'id': 6, 'name': 'Mariah' },{ 'id': 7, 'name': 'John' },{ 'id': 1, 'name': 'Louis' } ] c = Counter(x['name'] for x in my_list_of_dicts) duplicates = [x for x in my_list_of_dicts if c[x['name']] > 1]
Question: Is there a way to identify leading and trailing NAs in a pandas.DataFrame Currently I do the following but it seems not straightforward: import pandas as pd df = pd.DataFrame(dict(a=[0.1, 0.2, 0.2], b=[None, 0.1, None], c=[0.1, None, 0.1]) lead_na = (df.isnull() == False).cumsum() == 0 trail_na = (df.iloc[::-1].isnull() == False).cumsum().iloc[::-1] == 0 trail_lead_nas = top_na | trail_na  Any ideas how this could be expressed more efficiently? Answer: %timeit df.ffill().isna() | df.bfill().isna() The slowest run took 29.24 times longer than the fastest. This could mean that an intermediate result is being cached. 31 ms ± 25.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) %timeit ((df.isnull() == False).cumsum() == 0) | ((df.iloc[::-1].isnull() ==False).cumsum().iloc[::-1] == 0) 255 ms ± 66.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) 

Answer: How about this df.ffill().isna() | df.bfill().isna() Out[769]: a b c 0 False True False 1 False False False 2 False True False   df = pd.concat([df] * 1000, ignore_index=True) In [134]: %%timeit ...: lead_na = (df.isnull() == False).cumsum() == 0 ...: trail_na = (df.iloc[::-1].isnull() == False).cumsum().iloc[::-1] == 0 ...: trail_lead_nas = lead_na | trail_na ...: 11.8 ms ± 105 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [135]: %%timeit ...: df.ffill().isna() | df.bfill().isna() ...: 2.1 ms ± 50 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Question: I have the following pandas dataframe: df1 = pd.DataFrame({'date': [200101,200101,200101,200101,200102,200102,200102,200102],'blockcount': [1,1,2,2,1,1,2,2],'reactiontime': [350,400,200,250,100,300,450,400]})  I am trying to create a hierarchical dictionary, with the values of the embedded dictionary as lists, that looks like this: {200101: {1:[350, 400], 2:[200, 250]}, 200102: {1:[100, 300], 2:[450, 400]}}  How would I do this? The closest I get is using this code: df1.set_index('date').groupby(level='date').apply(lambda x: x.set_index('blockcount').squeeze().to_dict()).to_dict()  Which returns: {200101: {1: 400, 2: 250}, 200102: {1: 300, 2: 400}} 

Answer: Here is another way using pivot_table: d = df1.pivot_table(index='blockcount',columns='date', values='reactiontime',aggfunc=list).to_dict()   print(d) {200101: {1: [350, 400], 2: [200, 250]}, 200102: {1: [100, 300], 2: [450, 400]}}
Question: 100_year = date.today().year - age + 100 ^ SyntaxError: invalid decimal literal  I'm trying to understand what the problem is.

Answer: Python identifiers can not start with a number. The 'arrow' points to year because underscore is a valid thousands separator in Python >= 3.6, so 100_000 is a valid integer literal.
Question: I've recently installed both Pyenv and Poetry and want to create a new Python 3.8 project. I've set both the global and local versions of python to 3.8.1 using the appropriate Pyenv commands (pyenv global 3.8.1 for example). When I run pyenv version in my terminal the output is 3.8.1. as expected. Now, the problem is that when I create a new python project with Poetry (poetry new my-project), the generated pyproject.toml file creates a project with python 2.7: [tool.poetry] name = "my-project" version = "0.1.0" description = "" authors = ["user <user@email.com>"] [tool.poetry.dependencies] python = "^2.7" [tool.poetry.dev-dependencies] pytest = "^4.6" [build-system] requires = ["poetry>=0.12"] build-backend = "poetry.masonry.api"  It seems that Poetry defaults back to the system version of Python. How do I change this so that it uses the version installed with Pyenv? Edit I'm using MacOS, which comes bundled with Python 2.7. I think that might be causing some of the issues here. I've reinstalled Python 3.8 again with Pyenv, but when I hit Poetry install I get the following error: The currently activated Python version 2.7.16 is not supported by the project (^3.8). Trying to find and use a compatible version. [NoCompatiblePythonVersionFound] Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.  Should I create an environment explicitly for the project using Pyenv or should the project be able to access the correct Python version after running pyenv local 3.8.1.? When I do the latter, nothing changes and I still get the same errors.

Answer: Alright, I figured the problem. A little embarrassingly, I had not run pyenv shell 3.8.1 before running any of the other commands. Everything works now. Thank you all for your efforts.
Question: Given the following pandas data frame:   | a b --+----- 0 | 1 A 1 | 2 A 2 | 3 A 3 | 4 A 4 | 1 B 5 | 2 B 6 | 3 B 7 | 1 C 8 | 3 C 9 | 4 C  If you group it by column b I want to perform an action that keeps only the rows where they have column a in common. The result would be the following data frame:  | a b --+----- 0 | 1 A 2 | 3 A 4 | 1 B 6 | 3 B 7 | 1 C 8 | 3 C  Is there some built in method to do this?

Answer: You can try pivot_table with dropna here then filter using sreries.isin : s = df.pivot_table(index='a',columns='b',aggfunc=len).dropna().index df[df['a'].isin(s)]  Similarly with crosstab: s = pd.crosstab(df['a'],df['b']) df[df['a'].isin(s[s.all(axis=1)].index)]    a b 0 1 A 2 3 A 4 1 B 6 3 B 7 1 C 8 3 C
Question: I have a list containing thousands of sub-lists. Each of these sub-lists contain a combination of mixed strings and boolean values, for example: lst1 = [['k', 'b', False], ['k', 'a', True], ['a', 'a', 'a'], ['a', 'b', 'a'], ['a', 'a' , False], ...]  I want to sort this list in accordance with the contents of the sub-lists, like: lst2 = [['a', 'a', 'a'], ['a', 'a' , False], ['a', 'b', 'a'], ['k', 'a', True], ['k', 'b', False], ...]  I've tried sorting it like this: lst2 = sorted([list(sorted(x)) for x in lst1]) print(lst2)  This doesn't work because of the combination of boolean values with strings in some fields, so I get TypeError: '<' not supported between instances of 'bool' and 'str'. I've also tried a brute force method, creating every possible combination and then checking them to see if which are in the first list: col1 = ['a', 'b', 'c', d, e, f, g, h, i, j, k, ..., True, False] col2 = ['a', 'b', 'c', d, e, f, g, h, i, j, k, ..., True, False] col3 = ['a', 'b', 'c', d, e, f, g, h, i, j, k, ..., True, False] lst2 = list() for t1 in col1: for t2 in col2: for t3 in col3: test_sublist = [t1, t2, t3] if test_sublist in lst1: lst2.append(test_sublist)  This way works well enough, because I'm able to automatically create sorted lists for each column, col 1, col 2, and col 3, but it takes way too long to run (more than 3 days). Is there a better solution for sorting mixed string/boolean lists like these?

Answer: These handle any lengths, not just length 3. And bools in any places, not just the last column. For keying, they turn each element of each sublist into a tuple.  Solution 1: sorted(lst1, key=lambda s: [(e is False, e is True, e) for e in s])  Turns strings into (False, False, thestring) so they come first. Turns True into (False, True, True) so it comes next. Turns False into (True, False, False) so it comes last. Though I think of it the reverse way, as in "First deprioritize False, then deprioritize True". The general form is key=lambda x: (shall_come_last(x), x).  Solution 2: sorted(lst1, key=lambda s: [((e is True) + 2 * (e is False), e) for e in s])  Turns strings into (0, thestring) so they come first. Turns True into (1, True) so it comes next. Turns False into (2, False) so it comes last.  Solution 3: sorted(lst1, key=lambda s: [(0, e) if isinstance(e, str) else (2 - e,) for e in s])  Turns strings into (0, thestring) so they come first. Turns True into (1,) so it comes next. Turns False into (2,) so it comes last.
Question: I have a list of durations like below ['5d', '20h', '1h', '7m', '14d', '1m']  where d stands for days, h stands for hours and m stands for minutes. I want to get the highest duration from this list(14d in this case). How can I get that from this list of strings?

Answer: Pure python solution. We could store mapping between our time extensions (m, h, d) and minutes (here time_map), to find highest duration. Here we're using max() with key argument to apply our mapping. inp = ['5d', '20h', '1h', '7m', '14d', '1m'] time_map = {'m': 1, 'h': 60, 'd': 24*60} print(max(inp, key=lambda x:int(x[:-1])*time_map[x[-1]])) # -> 14d
Question: Is there a way to do an assignment only if the assigned value is not None, and otherwise do nothing? Of course we can do: x = get_value() if get_value() is not None  but this will read the value twice. We can cache it to a local variable: v = get_value() x = v if v is not None  but now we have made two statements for a simple thing. We could write a function: def return_if_not_none(v, default): if v is not None: return v else: return default  And then do x = return_if_not_none(get_value(), x). But surely there is already a Python idiom to accomplish this, without accessing x or get_value() twice and without creating variables? Put in another way, let's say =?? is a Python operator similar to the C# null coalesce operator. Unlike the C# ??=, our fictional operator checks if the right hand side is None: x = 1 y = 2 z = None x =?? y print(x) # Prints "2" x =?? z print(x) # Still prints "2"  Such a =?? operator would do exactly what my question is asking.

Answer: In python 3.8 you can do something like this if (v := get_value()) is not None: x = v  Updated based on Ryan Haining solution, see in comments