Package: cache-pandas
cache-pandasEasily cache outputs of functions that generate pandas DataFrames to file or in memory. Useful for data science projects where a large DataFrame is generated, but will not change for some time, so it could be cached to file. The next time the script runs, it will use the cached version.Caching pandas dataframes to csv filecache-pandasincludes the decoratorcache_to_csv, which will cache the result of a function (returning a DataFrame) to a csv file. The next time the function or script is run, it will take that cached file, instead of calling the function again.An optional expiration time can also be set. This might be useful for a webscraper where the output DataFrame may change once a day, but within the day, it will be the same. If the decorated function is called after the specified cache expiration, the DataFrame will be regenerated.ExampleThe following example will cache the resulting DataFrame tofile.csv. It will regenerate the DataFrame and its cache if the function is called a second time atleast 100 seconds after the first.fromcache_pandasimportcache_to_csv@cache_to_csv("file.csv",refresh_time=100)defsample_constant_function()->pd.DataFrame:"""Sample function that returns a constant DataFrame, for testing purpose."""data={"ints":list(range(NUM_SAMPLES)),"strs":[str(i)foriinrange(NUM_SAMPLES)],"floats":[float(i)foriinrange(NUM_SAMPLES)],}returnpd.DataFrame.from_dict(data)Argsfilepath: Filepath to save the cached CSV.
refresh_time: Time seconds. If the file has not been updated in longer than refresh_time, generate the file
    anew. If `None`, the file will never be regenerated if a cached version exists.
create_dirs: Whether to create necessary directories containing the given filepath.Caching pandas dataframes to memorycache-pandasincludes the decoratortimed_lru_cache, which will cache the result of a function (returning a DataFrame) to a memory, usingfunctools.lru_cache.An optional expiration time can also be set. This might be useful for a webscraper where the output DataFrame may change once a day, but within the day, it will be the same. If the decorated function is called after the specified cache expiration, the DataFrame will be regenerated.ExampleThe following example will cache the resulting DataFrame in memory. It will regenerate the DataFrame and its cache if the function is called a second time atleast 100 seconds after the first.fromcache_pandasimporttimed_lru_cache@timed_lru_cache(seconds=100,maxsize=None)defsample_constant_function()->pd.DataFrame:"""Sample function that returns a constant DataFrame, for testing purpose."""data={"ints":list(range(NUM_SAMPLES)),"strs":[str(i)foriinrange(NUM_SAMPLES)],"floats":[float(i)foriinrange(NUM_SAMPLES)],}returnpd.DataFrame.from_dict(data)Argsseconds: Number of seconds to retain the cache.
maxsize: Maximum number of items to store in the cache. See `functools.lru_cache` for more details.
typed: Whether arguments of different types will be cached separately. See `functools.lru_cache` for more details.Composingcache_to_csvandtimed_lru_cachecache_to_csvandtimed_lru_cachecan even be composed together. Usually the correct way to do this is to wraptimed_lru_cachelast, becausecache_to_csvwill always check the file before calling the function. The refresh time can even differ between the two caching mechanisms.Examplefromcache_pandasimporttimed_lru_cache,cache_to_csv@timed_lru_cache(seconds=100,maxsize=None)@cache_to_csv("file.csv",refresh_time=100)defsample_constant_function()->pd.DataFrame:"""Sample function that returns a constant DataFrame, for testing purpose."""data={"ints":list(range(NUM_SAMPLES)),"strs":[str(i)foriinrange(NUM_SAMPLES)],"floats":[float(i)foriinrange(NUM_SAMPLES)],}returnpd.DataFrame.from_dict(data)