eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to plot a pandas.DataFrame, but getting an unexplainable ValueError. Here is sample code causing the problem: import pandas as pd import matplotlib.pyplot as plt from io import StringIO import matplotlib.dates as mdates weekday_fmt = mdates.DateFormatter('%a %H:%M') test_csv = 'datetime,x1,x2,x3,x4,x5,x6\n' \ '2021-12-06 00:00:00,8,42,14,23,12,2\n' \ '2021-12-06 00:15:00,17,86,68,86,92,45\n' \ '2021-12-06 00:30:00,44,49,81,26,2,95\n' \ '2021-12-06 00:45:00,35,78,33,18,80,67' test_df = pd.read_csv(StringIO(test_csv), index_col=0) test_df.index = pd.to_datetime(test_df.index) plt.figure() ax = test_df.plot() ax.set_xlabel(f'Weekly aggregation') ax.set_ylabel('y-label') fig = plt.gcf() fig.set_size_inches(12.15, 5) ax.get_legend().remove() ax.xaxis.set_major_formatter(weekday_fmt) # This and the following line are the ones causing the issues ax.xaxis.set_minor_formatter(weekday_fmt) plt.show()  If the two formatting lines are removed, the code runs through, but if I leave them in there, I get a ValueError: ValueError: Date ordinal 27312480 converts to 76749-01-12T00:00:00.000000 (using epoch 1970-01-01T00:00:00), but Matplotlib dates must be between year 0001 and 9999. The reason seems to be that the conversion of datetime in pandas and matplotlib are incompatible. This could probably be circumvented by not using the built-in plot-function of pandas. Is there another way? Thanks! My package versions are: pandas 1.3.4 numpy 1.19.5 matplotlib 3.4.2 python 3.8.10 

Answer: Thanks to the comments by Jody Klymak and MrFuppes, I found the answer to simply be ax = test_df.plot(x_compat=True). For anybody stumbling upon this in future, here comes the full explanation of what is happening: When using the plot-function, pandas takes over the formatting of x-tick (and possibly other features). The selected x-tick-values shown to matplotlib do not need to correspond with what one would expect. In the shown example, the function ax.get_xlim() returns (27312480.0, 27312525.0). Using x_compat=True forces pandas to hand the correct values over to matplotlib where the formatting then happens. Since this was not clear to me from the error message I received, this post might help future viewers searching for that error message.
Question: The following code: from typing import Union def process(actions: Union[list[str], list[int]]) -> None: for pos, action in enumerate(actions): act(action) def act(action: Union[str, int]) -> None: print(action)  generates a mypy error:  Argument 1 to "act" has incompatible type "object"; expected "Union[str, int]" However when removing the enumerate function the typing is fine: from typing import Union def process(actions: Union[list[str], list[int]]) -> None: for action in actions: act(action) def act(action: Union[str, int]) -> None: print(action)  Does anyone know what the enumerate function is doing to effect the types? This is python 3.9 and mypy 0.921

Answer: enumerate.__next__ needs more context than is available to have a return type more specific than Tuple[int, Any], so I believe mypy itself would need to be modified to make the inference that enumerate(actions) produces Tuple[int,Union[str,int]] values. Until that happens, you can explicitly cast the value of action before passing it to act. from typing import Union, cast StrOrInt = Union[str, int] def process(actions: Union[list[str], list[int]]) -> None: for pos, action in enumerate(actions): act(cast(StrOrInt, action)) def act(action: Union[str, int]) -> None: print(action)  You can also make process generic (which now that I've thought of it, is probably a better idea, as it avoids the overhead of calling cast at runtime). from typing import Union, cast, Iterable, TypeVar T = TypeVar("T", str, int) def process(actions: Iterable[T]) -> None: for pos, action in enumerate(actions): act(action) def act(action: T) -> None: print(action)  Here, T is not a union of types, but a single concrete type whose identity is fixed by the call to process. Iterable[T] is either Iterable[str] or Iterable[int], depending on which type you pass to process. That fixes T for the rest of the call to process, which every call to act must take the same type of argument. An Iterable[str] or an Iterable[int] is a valid argument, binding T to int or str in the process. Now enumerate.__next__ apparently can have a specific return type Tuple[int, T].
Question: I have a python script that uses DigitalOcean tools (doctl and kubectl) I want to containerize. This means my container will need python, doctl, and kubectl installed. The trouble is, I figure out how to install both python and DigitalOcean tools in the dockerfile. I can install python using the base image "python:3" and I can also install the DigitalOcean tools using the base image "alpine/doctl". However, the rule is you can only use one base image in a dockerfile. So I can include the python base image and install the DigitalOcean tools another way: FROM python:3 RUN <somehow install doctl and kubectl> RUN pip install firebase-admin COPY script.py CMD ["python", "script.py"]  Or I can include the alpine/doctl base image and install python3 another way. FROM alpine/doctl RUN <somehow install python> RUN pip install firebase-admin COPY script.py CMD ["python", "script.py"]  Unfortunately, I'm not sure how I would do this. Any help in how I can get all these tools installed would be great!

Answer: just add this with any other thing you want to apt-get install: RUN apt-get update && apt-get install -y \ python3.6 &&\ python3-pip &&\  in alpine it should be something like: RUN apk add --update --no-cache python3 && ln -sf python3 /usr/bin/python &&\ python3 -m ensurepip &&\ pip3 install --no-cache --upgrade pip setuptools &&\
Question: Can the cursor.execute call below execute multiple SQL queries in one go? cursor.execute("use testdb;CREATE USER MyLogin")  I don't have python setup yet but want to know if above form is supported by cursor.execute? import pyodbc # Some other example server values are # server = 'localhost\sqlexpress' # for a named instance # server = 'myserver,port' # to specify an alternate port server = 'tcp:myserver.database.windows.net' database = 'mydb' username = 'myusername' password = 'mypassword' cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password) cursor = cnxn.cursor() #Sample select query cursor.execute("SELECT @@version;") row = cursor.fetchone() while row: print(row[0]) row = cursor.fetchone() 

Answer: Multiple SQL statements in a single string is often referred to as an "anonymous code block". There is nothing in pyodbc (or pypyodbc) to prevent you from passing a string containing an anonymous code block to the Cursor.execute() method. They simply pass the string to the ODBC Driver Manager (DM) which in turn passes it to the ODBC Driver. However, not all ODBC drivers accept anonymous code blocks by default. Some databases default to allowing only a single SQL statement per .execute() to protect us from SQL injection issues. For example, MySQL/Connector ODBC defaults MULTI_STATEMENTS to 0 (off) so if you want to run an anonymous code block you will have to include MULTI_STATEMENTS=1 in your connection string. Note also that changing the current database by including a USE … statement in an anonymous code block can sometimes cause problems because the database context changes in the middle of a transaction. It is often better to execute a USE … statement by itself and then continue executing other SQL statements.
Question: When using conda install, is it possible to install different packages from different channels in one line? For example could one do something like this? conda install -c <channel_1> <package_1> -c <channel_2> <package_2> ...?

Answer: The --channel argument The --channel, -c flag tells Conda where to search for packages, but does not necessarily constrain where a specific package should be sourced. Moreover, the order that channels are specified applies to the whole solving process, and has no contextual relationship with adjacent package specifications. For example, the following commands are all completely identical after parsing: conda install -c A -c B pkg1 pkg2 conda install -c A pkg1 -c B pkg2 conda install -c A pkg2 -c B pkg1 conda install pkg1 pkg2 -c A -c B  and all of these will prioritize channel A over B (under channel_priority: strict or flexible). It should be emphasized that this does not guarantee either of these channels will be used, only considered. I think it helps to have a prosaic translation of the above command:  With channels A and B prioritized, ensure that packages pkg1 and pkg2 are installed in the current environment.  Specifying channels per package However, the MatchSpec grammar is rather expressive and fully supports specifying the channel from which to source a given package. For example, if we want pkg1 from channel A and pkg2 from channel B, this would be expressed as: conda install A::pkg1 B::pkg2  which translates to the imperative:  Ensure that pkg1 from channel A and pkg2 from channel B are installed in the current environment.  Note that we don't even need to include the channel via a --channel argument, because the package specification itself indicates the channel. One only needs to include the --channel, -c argument if they want to source additional packages (e.g., dependencies of pkg1) from the additional channel.
Question: Is it possible to do method chaining in pandas when  no variable refering to the dataframe has been assigned, yet AND the method needs to refer to the dataframe?  Example: here data frame can be referred to by variable name. df = pd.DataFrame({"a":[1,2,3], "b":list("abc")}) df = (df .drop(df.tail(1).index) #.other_methods #... ) df  Is it possible to do this without having assigned the dataframe to a variable name? df = (pd.DataFrame({"a":[1,2,3], "b":list("abc")}) .drop(??.tail(1).index) #.other_methods #... ) df  Thanks!

Answer: You need some reference to the dataframe in order to use it in multiple independent places. That means binding a reusable name to the value returned by pd.DataFrame. A "functional" way to create such a binding is to use a lambda expression instead of an assignment statement. df = (lambda df: df.drop(df.tail(1).index)....)(pd.DataFrame(...))  The lambda expression defines some function that uses whatever value is passed as an argument as the value of the name df; you then immediately call that function on your original dataframe.
Question: In my Enum, i have defined a classmethod for coercing a given value to an Enum member. The given value may already be an instance of the Enum, or it may be a string holding an Enum value. In order to decide whether it needs conversion, i check if the argument is an instance of the class, and only pass it on to int() if it is not. At which point – according to the type hints for the argument 'item' – it must be a string. The class look like this: T = TypeVar('T', bound='MyEnum') class MyEnum(Enum): A = 0 B = 1 @classmethod def coerce(cls: Type[T], item: Union[int, T]) -> T: return item if isinstance(item, cls) else cls(int(item))  mypy fails with:  error: Argument 1 to "int" has incompatible type "Union[str, T]"; expected "Union[str, bytes, SupportsInt, SupportsIndex, _SupportsTrunc]"  Why?

Answer: It's because int cannot be constructed from Enum. From documentation of int(x)  If x is not a number or if base is given, then x must be a string, bytes, or bytearray instance representing an integer literal in radix base  So to make it work you can make your Enum inherit from str class MyEnum(str, Enum): A = 0 B = 1  Or use IntEnum class MyEnum(IntEnum): A = 0 B = 1  Edit: The question why isinstance doesn't narrow T type enough still left. Couldn't find any prove of my theory, but my (hopefully logical) explanation is - T is bounded to MyEnum so it's MyEnum and any subclass of it. If we have subclass scenario, isinstance(item, MyEnumSubclass) won't exclude case when item is just MyEnum class. If we use MyEnum directly in typing, problem will disappear.  @classmethod def coerce(cls: t.Type["MyEnum"], item: t.Union[int, T]) -> "MyEnum": return item if isinstance(item, cls) else cls(int(item))
Question: Why is there a difference between these?: # Python f = open("./text.txt", "r") for i in f.readlines(): for l in i: print(print(l == "\n", ":", l)) f.close() # ----------------------------- # Julia f = open("./text.txt", "r") while !eof(f) for l in readline(file) println(l == '\n', " : ", l) end end close(f)  The Python one outputs this: False : h False : e False : l False : l False : o False : False : W False : o False : r False : l False : d True : <--- yep, it is as expected False : y False : a False : y  The Julia one outputs this: false : h false : e false : l false : l false : o false : <--- is this not a \n?? false : W false : o false : r false : l false : d false : y false : a false : y  text.txt is this: hello World yay  As you can see the outputs are different. How can I make the Julia one behaves like the Python one? Are there other ways of reading a file in Julia?

Answer: You can do this: # Julia f = open("./text.txt", "r") while !eof(f) for l in readline(file, keep=true) println(l == '\n', " : ", l) end end close(f)  By default, it discards \n's, but you can keep them by adding keep=true.
Question: I'm training a CNN model on images. Initially, I was training on image patches of size (256, 256) and everything was fine. Then I changed my dataloader to load full HD images (1080, 1920) and I was cropping the images after some processing. In this case, the GPU memory keeps increasing with every batch. Why is this happening? PS: While tracking losses, I'm doing loss.detach().item() so that loss is not retained in the graph.

Answer: As suggested here, deleting the input, output and loss data helped. Additionally, I had the data as a dictionary. Just deleting the dictionary isn't sufficient. I had to iterate over the dict elements and delete all of them.
Question: Say I have the 2 Dataframes below; one with a list of students and test scores, and different student sessions that made up of the students. Say I want to add a new column, "Sum", to df with the sum of the scores for each session and a new column for the number of years passed since the most recent year that either student took the test, "Years Elapsed". What is the best way to accomplish this? I can make the students a class and make each student an object but then I am stuck on how to link the object to their name in the dataframe. data1 = {'Student': ['John','Kim','Adam','Sonia'], 'Score': [92,100,76,82], 'Year': [2015,2013,2016,2018]} df_students = pd.DataFrame(data1, columns=['Student','Score','Year']) data2 = {'Session': [1,2,3,4], 'Student1': ['Sonia','Kim','John','Adam'], 'Student2': ['Adam','Sonia','Kim','John']} df = pd.DataFrame(data2, columns=['Session','Student1','Student2'])  The desired outcome: outcome = {'Session': [1,2,3,4], 'Student1': ['Sonia','Kim','John','Adam'], 'Student2': ['Adam','Sonia','Kim','John'], 'Sum': [158, 182, 192, 168], 'Years Elapsed': [4,4,7,6]} df_outcome = pd.DataFrame(outcome, columns=['Session','Student1','Student2','Sum','Years Elasped'])  I have made a class called Student and made each student an object but after this is where I am stuck. df_students.columns = df_students.columns.str.lower() class Student: def __init__(self, s, sc, yr): self.student = s self.score = sc self.year = yr students = [Student(row.student, row.score, row.year) for index, row in df_students.iterrows()] #check to see if list of objects was created correctly s1 = students[1] s1.__dict__  Thanks in advance!

Answer: Using apply method: import pandas as pd data1 = {'Student': ['John','Kim','Adam','Sonia'], 'Score': [92,100,76,82], 'Year': [2015,2013,2016,2018]} df_students = pd.DataFrame(data1, columns=['Student','Score','Year']) data2 = {'Session': [1,2,3,4], 'Student1': ['Sonia','Kim','John','Adam'], 'Student2': ['Adam','Sonia','Kim','John']} df = pd.DataFrame(data2, columns=['Session','Student1','Student2']) # SOLUTION def sum_scores(student1, student2): _score_s1 = df_students.loc[(df_students['Student']==student1)]['Score'].values[0] _score_s2 = df_students.loc[(df_students['Student']==student2)]['Score'].values[0] return _score_s1 + _score_s2 def years_elapsed(student1, student2): _year = pd.to_datetime("today").year _year_s1 = df_students.loc[(df_students['Student']==student1)]['Year'].values[0] _year_s2 = df_students.loc[(df_students['Student']==student2)]['Year'].values[0] return _year - max(_year_s1, _year_s2) df['sum_score'] = df.apply(lambda row: sum_scores(row['Student1'], row['Student2']), axis=1) df['years_elapsed'] = df.apply(lambda row: years_elapsed(row['Student1'], row['Student2']), axis=1) df
Question: so - I have built a bit of a rules engine in python - but I'm fairly new to python... my engine is fairly nice to use - but adding a new rule is pretty ugly, and I'm wondering if there's a way to clean it up. The key thing to remember is that rules have side-effects, rules can be combined with ands, ors, etc - and you only apply the side effects if the whole rule succeeded - ie the check if the rule succeeded can't be combined with perfoming the side effect. So every rule ends up looking something like this:  def sample_rule(): def check( item ): if item.doesnt_pass_some_condition(): return None def action_to_perform(): item.set_some_value() item.set_some_other_value() return action_to_perform return Rule(check)  which seems horribly ugly - but you don't seem to be able to do multiline lambdas or zero line lambas... I guess I'm looking for something like:  def sample_rule(): return Rule( lambda x: x.passes_condition(), lambda x: {x.set_some_value(), x.set_some_other_value)}  but both the condition and the side effect could be multiple lines, and the side effect is often empty. so is there a simpler pattern that i can apply that will apply to every case? (I really don't want to use the above pattern when I have exactly one line of condition and one line of side effect, and a completely different pattern in the other cases) just out of interest, at the end you end up with something like  rule1 = sample_rule().andalso( other_rule_1().or(other_rule_2)).butnot( other_rule_3) ... ... for thing_to_check in lots_of_things: for rule in lots_of_rules: if rule.apply_to( thing_to_check): break # take the first rule that applies 

Answer: Instead of defining multi-line lambdas (which python doesn't allow), you could define multiple lambdas in a list and then use all lambdas in the list as required: class Rule: def __init__(self, checks=None, actions=None): self.checks = checks if checks else [] self.actions = actions if actions else [] def apply_to(self, item): if all([check(item) for check in self.checks]): return self.actions else: return None sample_rule = Rule(checks=[lambda x: x.passes_condition()], actions=[lambda x: x.set_some_value(), lambda x: x.set_some_other_value()]) # simpler rule when no actions/side effects are needed simple_rule = Rule(checks=[lambda x: x.passes_condition()])  Giving checks and actions a default value of None in the class also means you don't have to define them if there are no checks/actions for a given rule; further simplifying the definition of new rules.
Question: I am trying to use recursion to solve the OddOccurrencesInArray Problem in Codility, in which  we are given an array with N elements, N is always odd all of the elements of the array except for one has a total even number of occurrences we need to write code that returns the one unpaired value  For example, if the array given is [9, 3, 9, 3, 7, 9, 9], the code must return 7, because that is the only element in the array which is unpaired. My solution pseudocode/thought process was:  sort the array if the first two elements are equal to each other, remove them and run the solution algorithm again recursively on the array minus the first two elements (after sorting) i.e. if the unpaired element is not found, we keep reducing the size of the array if the first two elements are NOT equal to each other, the first element of the array must be the unpaired item  My implementation was: def solution(A): # write your code in Python 3.6 if len(A) > 1: A = sorted(A) if A[0] != A[1]: return A[0] else: solution(A[2:]) else: return A[0]  I keep getting the error message Invalid result type, int expected, <class 'NoneType'> found. RUNTIME ERROR (tested program terminated with exit code 1) Can anyone help me figure out what this means and how I can correct it? Algorithmically, I think my solution is sound, and I don't understand why it isn't returning the integer values as I specified.

Answer: I would suggest a different approach altogether. A recursive approach is not incorrect, however repeated calls to sorted is highly inefficient, especially if the input is significantly large. def solve(t): s = set() for v in t: s.add(v) if v not in s else s.remove(v) return list(s)  input = [9, 3, 9, 3, 7, 9, 9] solve(input)  We can visualize s over the course of the evaluation - {} # <- initial s {9} # <- 9 is added {9,3} # <- 3 is added {3} # <- 9 is removed {} # <- 3 is removed {7} # <- 7 is added {7,9} # <- 9 is added {7} # <- 9 is removed  The finally list(s) is returned converting {7} to [7]. To output the answer we can write a simple if/elif/else - unpaired = solve(input) if (len(unpaired) < 1): print("there are no unpaired elements") elif (len(unpaired) > 1): print("there is more than one unpaired element") else: print("answer:", unpaired[0])  Another option is to have solve return the first unpaired element or None - def solve(t): s = set() for v in t: s.add(v) if v not in s else s.remove(v) for v in s: return v # <- immediately return first element  answer = solve(input) if answer is None: print("no solution") else: print("the solution is", answer)
Question: If have some code like this: class Foo(): def open(self, bar): # Doing some fancy stuff here, i.e. opening "bar" pass  When I run flake8 with the flake8-builtins plug-in I get the error A003 class attribute "open" is shadowing a python builtin  I don't understand how the method could possibly shadow the built-in open-function, because the method can only be called using an instance (i.e. self.open("") or someFoo.open("")). Is there some other way code expecting to call the built-in ends up calling the method? Or is this a false positive of the flake8-builtins plug-in?

Answer: Not really a practical case, but your code would fail if you wanted to use the built-it functions on the class level after your shadowed function has been initialized: class Foo: def open(self, bar): pass with open('myfile.txt'): print('did I get here?') >>> TypeError: open() missing 1 required positional argument: 'bar'  The same would also be true with other built-in functions, such as print class Foo: def print(self, bar): pass print('did I get here?') >>> TypeError: print() missing 1 required positional argument: 'bar'