Package: wvutils
WVUtilsWVUtilsis a collection of utilities that are shared across multiplePhosmicprojects.Requirements:WVUtilsrequires Python 3.10 or higher and is platform independent.Issue reportingIf you discover an issue with WVUtils, please report it athttps://github.com/Phosmic/wvutils/issues.LicenseThis program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.You should have received a copy of the GNU General Public License along with this program. If not, seehttps://www.gnu.org/licenses/.RequirementsInstallingLibraryInstallingMost stable version fromPyPi:python3-mpipinstallwvutilsDevelopment version fromGitHub:gitclonegit+https://github.com/Phosmic/wvutils.gitcdwvutils
python3-mpipinstall-e.Librarywvutils.awsUtilities for interacting with AWS services.This module provides utilities for interacting with AWS services.get_boto3_sessiondefget_boto3_session(region_name:AWSRegion)->SessionGet the globally shared Boto3 session for a region (thread-safe).Todo:Add support for other session parameters.Arguments:region_nameAWSRegion- Region name for the session.Returns:Session- Boto3 session.clear_boto3_sessionsdefclear_boto3_sessions()->intClear all globally shared Boto3 sessions (thread-safe).Returns:int- Number of sessions cleared.boto3_client_ctx@contextmanagerdefboto3_client_ctx(service_name:str,region_name:AWSRegion)Context manager for a Boto3 client (thread-safe).Todo:Add support for other session parameters.Arguments:service_namestr- Name of the service.region_nameAWSRegion- Region name for the service.Yields:Any- Boto3 client.Raises:ClientError- If an error occurs.parse_s3_uridefparse_s3_uri(s3_uri:str)->tuple[str,str]Parse the bucket name and path from a S3 URI.Arguments:s3_uristr- S3 URI to parse.Returns:tuple[str, str]- Bucket name and path.download_from_s3defdownload_from_s3(file_path:FilePath,bucket_name:str,bucket_path:str,region_name:AWSRegion,overwrite:bool=False)->NoneDownload a file from S3.Arguments:file_pathFilePath- Output path to use while downloading the file.bucket_namestr- Name of the S3 bucket containing the file.bucket_pathstr- Path of the S3 bucket containing the file.region_nameAWSRegion- Region name for S3.overwritebool- Overwrite file on disk if already exists. Defaults to False.Raises:FileExistsError- If the file already exists and overwrite is False.upload_file_to_s3defupload_file_to_s3(file_path:FilePath,bucket_name:str,bucket_path:str,region_name:AWSRegion)->NoneUpload a file to S3.Arguments:file_pathFilePath- Path of the file to upload.bucket_namestr- Name of the S3 bucket to upload the file to.bucket_pathstr- Path in the S3 bucket to upload the file to.region_nameAWSRegion- Region name for S3.Raises:FileNotFoundError- If the file does not exist.upload_bytes_to_s3defupload_bytes_to_s3(raw_b:bytes,bucket_name:str,bucket_path:str,region_name:AWSRegion)->NoneWrite bytes to a file in S3.Arguments:raw_bbytes- Bytes of the file to be written.bucket_namestr- Name of the S3 bucket to upload the file to.bucket_pathstr- Path in the S3 bucket to upload the file to.region_nameAWSRegion- Region name for S3.secrets_fetchdefsecrets_fetch(secret_name:str,region_name:AWSRegion)->str|int|float|list|dict|NoneRequest and decode a secret from Secrets.Arguments:secret_namestr- Secret name to use.region_nameAWSRegion- Region name for Secrets.Returns:str | int | float | list | dict | None- Secret string.Raises:ClientError- If an error occurs while fetching the secret.ValueError- If the secret is not valid JSON.athena_execute_querydefathena_execute_query(query:str,database_name:str,region_name:AWSRegion)->str|NoneExecute a query in Athena.Arguments:querystr- Query to execute.database_namestr- Name of database to execute the query against.region_nameAWSRegion- Region name for Athena.Returns:str | None- Query execution ID of the query.athena_retrieve_querydefathena_retrieve_query(qeid:str,database_name:str,region_name:AWSRegion)->str|NoneRetrieve the S3 URI for results of a query in Athena.Arguments:qeidstr- Query execution ID of the query to fetch.database_namestr- Name of the database the query is running against (for debugging).region_nameAWSRegion- Region name for Athena.Returns:str | None- Current status of the query, or S3 URI where results are stored.Raises:ValueError- If the query execution ID is unknown or missing.athena_stop_querydefathena_stop_query(qeid:str,region_name:AWSRegion)->NoneStop the execution of a query in Athena.Arguments:qeidstr- Query execution ID of the query to stop.region_nameAWSRegion- Region name for Athena.wvutils.errorsCustom errors.This module contains custom exceptions that are used throughout the package.WVUtilsErrorObjectsclassWVUtilsError(Exception)Base class for all wvutils exceptions.JSONErrorObjectsclassJSONError(Exception)Base class for all JSON exceptions.JSONEncodeErrorObjectsclassJSONEncodeError(JSONError,TypeError)Raised when JSON serializing fails.JSONDecodeErrorObjectsclassJSONDecodeError(JSONError,ValueError)Raised when JSON deserializing fails.PickleErrorObjectsclassPickleError(Exception)Base class for all pickle exceptions.PickleEncodeErrorObjectsclassPickleEncodeError(PickleError,TypeError)Raised when pickle serializing fails.PickleDecodeErrorObjectsclassPickleDecodeError(PickleError,ValueError)Raised when unpickling fails.HashErrorObjectsclassHashError(Exception)Base class for all hashing exceptions.HashEncodeErrorObjectsclassHashEncodeError(HashError,TypeError)Raised when hashing fails.wvutils.pathUtilities for working with paths.This module provides utilities for working with paths.is_pathlikedefis_pathlike(potential_path:Any)->boolCheck if an object is path-like.An object is path-like if it is a string or has a__fspath__method.Arguments:potential_pathAny- Object to check.Returns:bool- True if the object is path-like, otherwise False.stringify_pathdefstringify_path(file_path:FilePath)->strStringify a path-like object.The path-like object is first converted to a string, then the user directory is expanded.An object is path-like if it is a string or has a__fspath__method.Arguments:file_pathFilePath- Path-like object to stringify.Returns:str- Path-like object as a string.Raises:TypeError- If the object is not path-like.ensure_abspathdefensure_abspath(file_path:str)->strMake a path absolute if it is not already.Arguments:file_pathstr- Path to ensure is absolute.Returns:str- Absolute path.resolve_pathdefresolve_path(file_path:FilePath)->strStringify and resolve a path-like object.The path-like object is first converted to a string, then the user directory is expanded, and finally the path is resolved to an absolute path.An object is path-like if it is a string or has a__fspath__method.Arguments:file_pathFilePath- Path-like object to resolve.Returns:str- Absolute path of the path-like object as a string.Raises:TypeError- If the object is not path-like.xdg_cache_pathdefxdg_cache_path()->strBase directory to store user-specific non-essential data files.This should be '${HOME}/.cache', but the 'HOME' environment variable may not exist on non-POSIX-compliant systems.
On POSIX-compliant systems, the XDG base directory specification is followed exactly since '~' expands to '$HOME' if it is present.Returns:str- Path for XDG cache.wvutils.proxiesUtilities for working with proxies.This module provides utilities for working with proxies.ProxyManagerObjectsclassProxyManager()Manages a list of proxies.This class manages a list of proxies, allowing for randomization, re-use, etc.ProxyManager.add_proxiesdefadd_proxies(proxies:list[str],include_duplicates:bool=False)->NoneAdd additional proxy addresses.All proxy addresses added will be added to the end of the list.Arguments:proxieslist[str]- List of proxy addresses.include_duplicatesbool, optional- Whether to include duplicates. Defaults to False.ProxyManager.set_proxiesdefset_proxies(proxies:list[str])->NoneSet the proxy addresses.Note: This will clear all existing proxies.Arguments:proxieslist[str]- List of proxy addresses.ProxyManager.can_cycle@propertydefcan_cycle()->boolCheck if can cycle to the next proxy address.Returns:bool- True if can cycle, False otherwise.ProxyManager.cycledefcycle()->NoneAttempt to cycle to the next proxy address.ProxyManager.proxy@propertydefproxy()->str|NoneCurrent proxy address.Returns:str | None- Current proxy, or None if no proxies.https_to_httpdefhttps_to_http(address:str)->strConvert a HTTPS proxy address to HTTP.Arguments:addressstr- HTTPS proxy address.Returns:str- HTTP proxy address.Raises:ValueError- If the address does not start with 'http://' or 'https://'.prepare_http_proxy_for_requestsdefprepare_http_proxy_for_requests(address:str)->dict[str,str]Prepare a HTTP(S) proxy address for use with the 'requests' library.Arguments:addressstr- HTTP(S) proxy address.Returns:dict[str, str]- Dictionary of HTTP and HTTPS proxy addresses.Raises:ValueError- If the address does not start with 'http://' or 'https://'.wvutils.argsUtilities for parsing arguments from the command line.This module provides utilities for parsing arguments from the command line.nonempty_stringdefnonempty_string(name:str)->Callable[[str],str]Ensure a string is non-empty.Example:subparser.add_argument("hashtag",type=nonempty_string("hashtag"),help="A hashtag (without #)",)Arguments:namestr- Name of the function, used for debugging.Returns:Callable[[str], str]- The decorated function.safechars_stringdefsafechars_string(name:str,allowed_chars:Collection[str]|None=None)->Callable[[str],str]Ensure a string contains only safe characters.Example:subparser.add_argument("--session-key",type=safechars_string,help="Key to share a single token across processes",)Arguments:namestr- Name of the function, used for debugging.allowed_charsCollection[str] | None, optional- Custom characters used to validate the function name. Defaults to None.Returns:Callable[[str], str]- The decorated function.Raises:ValueError- If empty collection of allowed characters is provided.wvutils.generalGeneral utilities for working with Python.This module provides general utilities for working with Python.is_readable_iolikedefis_readable_iolike(potential_io:Any)->boolCheck if an object is a readable IO-like.An object is readable IO-like if it has one of the following:callablereadablemethod that returns TrueOr if it has all of the following:callablereadmethod.string attributemodethat contains "r".callableseekmethod.callableclosemethod.callable__enter__method.callable__exit__method.Arguments:potential_ioAny- Object to check.Returns:bool- True if the object is a readable IO-like, otherwise False.is_writable_iolikedefis_writable_iolike(potential_io:Any)->boolCheck if an object is a writable IO-like.An object is writable IO-like if it has one of the following:callablewritablemethod that returns TrueOr if it has all of the following:callablewritemethod.string attributemodethat contains "w".callableseekmethod.callableclosemethod.callable__enter__method.callable__exit__method.Arguments:potential_ioAny- Object to check.Returns:bool- True if the object is a writable IO-like, otherwise False.is_iolikedefis_iolike(potential_io:Any)->boolCheck if an object is IO-like.An object is IO-like if it has one of the following:io.IOBasebase classCallingis_readable_iolikereturns True. (seeis_readable_iolike)Callingis_writable_iolikereturns True. (seeis_writable_iolike)Arguments:potential_ioAny- Object to check.Returns:bool- True if the object is IO-like, otherwise False.count_lines_in_filedefcount_lines_in_file(file_path:FilePath)->intCount the number of lines in a file.Notes:All files have at least 1 line (# of lines = # of newlines + 1).Arguments:file_pathFilePath- Path of the file to count lines in.Returns:int- Total number of lines in the file.sys_set_recursion_limitdefsys_set_recursion_limit()->NoneRaise recursion limit to allow for more recurse.gc_set_thresholddefgc_set_threshold()->NoneReduce Number of GC Runs to Improve PerformanceNotes:Only applies to CPython.chunkerdefchunker(seq:Sequence[Any],n:int)->Generator[Sequence[Any],None,None]Iterate a sequence in sizenchunks.Arguments:seqSequence[Any]- Sequence of values.nint- Number of values per chunk.Yields:Sequence[Any]- Chunk of values with length <= n.Raises:ValueError- Ifnis 0 or negative.is_iterabledefis_iterable(obj:Any)->boolCheck if an object is iterable.Arguments:objAny- Object to check.Returns:bool- Whether the object is iterable.rename_keydefrename_key(obj:dict,src_key:str,dest_key:str,in_place:bool=False)->dict|NoneRename a dictionary key.Todo:Add support for nested keys.Add support for renaming multiple keys at once.Add support for non-string (built-in) key types.All of the following are True:isinstance(True,bool)isinstance(True,int)1==True1in{1:"a"}Truein{1:"a"}1in{True:"a"}Truein{True:"a"}1in{1:"a",True:"b"}Truein{1:"a",True:"b"}Arguments:objdict- Reference to the dictionary to modify.srcstr- Name of the key to rename.deststr- Name of the key to change to.in_placebool, optional- Perform in-place using the provided reference. Defaults to False.Returns:dict | None- Copy of the dictionary if in_place is False, otherwise None.unnest_keydefunnest_key(obj:dict,*keys:str)->Any|NoneFetch a value from a deeply nested dictionary.Arguments:objdict- Dictionary to recursively iterate.*keysstr- Ordered keys to fetch.Returns:Any | None- The result of the provided keys, or None if any key is not found.sort_dict_by_keydefsort_dict_by_key(obj:dict,reverse:bool=False,deep_copy:bool=False)->dict|NoneSort a dictionary by key.Arguments:objdict- Dictionary to sort.reversebool, optional- Sort in reverse order. Defaults to False.deep_copybool, optional- Return a deep copy of the dictionary. Defaults to False.Returns:dict | None- Dictionary sorted by key. Ifin_placeis True, None is returned.Raises:ValueError- If the dictionary keys are not of the same type.dedupe_listdefdedupe_list(values:list[Any],raise_on_dupe:bool=False)->list[Any]Remove duplicate values from a list.Example:dedupe_list([1,2,3,1,2,3])# [1, 2, 3]Arguments:valueslist[Any]- List of values to dedupe.raise_on_dupebool, optional- Raise an error if a duplicate is found. Defaults to False.Returns:list[Any]- List of unique values.Raises:ValueError- If a duplicate is found andraise_on_dupeis True.dupe_in_listdefdupe_in_list(values:list[Any])->boolCheck if a list has duplicate values.Arguments:valueslist[Any]- List of values to check.Returns:bool- Whether the list has duplicate values.invert_dict_of_strdefinvert_dict_of_str(obj:dict[Any,str],deep_copy:bool=False,raise_on_dupe:bool=False)->dictInvert a dictionary of strings.Notes:The value of the last key with a given value will be used.Example:invert_dict_of_str({"a":"b","c":"d"})# {"b": "a", "d": "c"}Arguments:objdict[Any, str]- Dictionary to invert.deep_copybool, optional- Return a deep copy of the dictionary. Defaults to False.raise_on_dupebool, optional- Raise an error if a duplicate is found. Defaults to False.Returns:dict- Inverted dictionary.Raises:ValueError- If a duplicate is found andraise_on_dupeis True.get_all_subclassesdefget_all_subclasses(cls:type)->list[type]Get all subclasses of a class.Arguments:clstype- Class to get subclasses of.Returns:list[type]- List of subclasses.wvutils.dtDatetime utilities.This module contains functions and classes that are used to work with datetimes.dtformatsObjectsclassdtformats()Datetime formats.Attributes:twitterstr- Twitter datetime format.redditstr- Reddit datetime format.generalstr- General datetime format.dbstr- Database datetime format.datestr- Date format.time_12hstr- 12-hour time format with timezone.time_24hstr- 24-hour time format with timezone.num_days_in_monthdefnum_days_in_month(year:int,month:int)->intDetermine the number of days in a month.Arguments:yearint- Year to check.monthint- Month to check.Returns:int- Number of days in the month.wvutils.parquetParquet utilities.This module provides utilities for working with Parquet files.get_parquet_sessiondefget_parquet_session(use_s3:bool,region_name:AWSRegion|None=None)->fs.FileSystemGet the globally shared Parquet filesystem session.Todo:Add support for other session parameters.Arguments:use_s3bool- Use S3 if True, otherwise uses local filesystem.region_nameAWSRegion | None, optional- AWS region name. Defaults to None.Returns:pyarrow.fs.FileSystem- The filesystem session.clear_parquet_sessionsdefclear_parquet_sessions()->intClear all globally shared Parquet filesystem sessions.Returns:int- Number of sessions cleared.create_pa_schemadefcreate_pa_schema(schema_template:dict[str,str])->pa.schemaCreate a parquet schema from a template.Example:{"key_a":"string","key_b":"integer","key_c":"float","key_d":"bool","key_e":"timestamp[s]","key_f":"timestamp[ms]","key_g":"timestamp[ns]",}becomespa.schema([("key_a",pa.string()),("key_b",pa.int64()),("key_c",pa.float64()),("key_d",pa.bool_()),("key_e",pa.timestamp("s",tz=utc)),("key_f",pa.timestamp("ms",tz=utc)),("key_g",pa.timestamp("ns",tz=utc)),])Arguments:schema_templateSequence[Sequence[str]]- Data names and parquet types for creating the schema.Returns:pa.schema- Final parquet schema.Raises:ValueError- If an unknown type name is encountered.force_dataframe_dtypesdefforce_dataframe_dtypes(dataframe:pd.DataFrame,template:dict[str,str])->pd.DataFrameForce the data types of a dataframe using a template.Arguments:dataframepd.DataFrame- Dataframe to force types.templatedict[str, str]- Template to use for forcing types.Returns:pd.DataFrame- Dataframe with forced types.Raises:ValueError- If an unknown type name is encountered.export_datasetdefexport_dataset(data:list[dict]|deque[dict]|pd.DataFrame,output_location:str|FilePath,primary_template:dict[str,str],partitions_template:dict[str,str],*,basename_template:str|None=None,use_s3:bool=False,region_name:AWSRegion|None=None,use_threads:bool=False,overwrite:bool=False)->NoneWrite to dataset to local filesystem or AWS S3.Arguments:datalist[dict] | deque[dict] | pd.DataFrame]- List, deque, or dataframe of objects to be written.output_locationstr | FilePath- Location to write the dataset to.primary_templatedict[str, str]- Parquet schema template to use for the table (excluding partitions).
partitions_template(dict[str, str]): Parquet schema template to use for the partitions.basename_templatestr, optional- Filename template to use when writing to S3 or locally. Defaults to None.use_s3bool, optional- Use S3 as the destination when exporting parquet files. Defaults to False.region_nameAWSRegion, optional- AWS region to use when exporting to S3. Defaults to None.use_threadsbool, optional- Use multiple threads when exporting. Defaults to False.overwritebool, optional- Overwrite existing files. Defaults to False.wvutils.restructUtilities for restructuring data.This module provides utilities for restructuring data, including serialization and hashing.JSONPythonJSONdictobjectlist, tuplearraystrstringint, float, int- & float-derived enumsnumberTruetrueFalsefalseNonenullHashNo content.PickleAn important difference between cloudpickle and pickle is that cloudpickle can serialize a function or class by value, whereas pickle can only serialize it by reference.
Serialization by reference treats functions and classes as attributes of modules, and pickles them through instructions that trigger the import of their module at load time.
Serialization by reference is thus limited in that it assumes that the module containing the function or class is available/importable in the unpickling environment.
This assumption breaks when pickling constructs defined in an interactive session, a case that is automatically detected by cloudpickle, that pickles such constructs by value.Read more:https://github.com/cloudpipe/cloudpickle/blob/master/README.md#overriding-pickles-serialization-mechanism-for-importable-constructsjson_dumpsdefjson_dumps(obj:JSONSerializable)->strEncode an object as JSON.Arguments:objJSONSerializable- Object to encode.Returns:str- Object encoded as JSON.Raises:JSONEncodeError- If the object could not be encoded.jsonl_dumpsdefjsonl_dumps(objs:Iterable[JSONSerializable])->strEncode objects as JSONL.Arguments:objsIterable[JSONSerializable]- Objects to encode.Returns:str- Objects encoded as JSONL.Raises:JSONEncodeError- If the objects could not be encoded.json_dumpdefjson_dump(obj:JSONSerializable,file_path:str)->NoneEncode an object as JSON and write it to a file.Arguments:file_pathstr- Path of the file to open.Raises:JSONEncodeError- If the object could not be encoded.jsonl_dumpdefjsonl_dump(objs:Iterable[JSONSerializable],file_path:str)->NoneEncode objects as JSONL and write them to a file.Arguments:objsIterable[JSONSerializable]- Objects to encode.file_pathstr- Path of the file to open.Raises:JSONEncodeError- If the objects could not be encoded.json_loadsdefjson_loads(encoded_obj:str)->JSONSerializableDecode a JSON-encoded object.Arguments:encoded_objstr- Object to decode.Returns:JSONSerializable- Decoded object.Raises:JSONDecodeError- If the object could not be decoded.json_loaddefjson_load(file_path:FilePath)->JSONSerializableDecode a file containing a JSON-encoded object.Arguments:file_pathFilePath- Path of the file to open.Returns:JSONSerializable- Decoded object.Raises:JSONDecodeError- If the file could not be decoded.jsonl_loaderdefjsonl_loader(file_path:FilePath,*,allow_empty_lines:bool=False)->Generator[JSONSerializable,None,None]Decode a file containing JSON-encoded objects, one per line.Arguments:file_pathFilePath- Path of the file to open.allow_empty_linesbool, optional- Whether to allow (skip) empty lines. Defaults to False.Yields:JSONSerializable- Decoded object.Raises:JSONDecodeError- If the line could not be decoded, or if an empty line was found andallow_empty_linesis False.squeegee_loaderdefsqueegee_loader(file_path:FilePath)->Generator[JSONSerializable,None,None]Automatically decode a file containing JSON-encoded objects.Supports multiple formats (JSON, JSONL, JSONL of JSONL, etc).Todo:Add support for pretty-printed JSON that has been appended to a file.Arguments:file_pathFilePath- Path of the file to open.Yields:JSONSerializable- Decoded object.Raises:JSONDecodeError- If the line could not be decoded.gen_hashdefgen_hash(obj:MD5Hashable)->strCreate an MD5 hash from a hashable object.Note: Tuples and deques are not hashable, so they are converted to lists.Arguments:objMD5Hashable- Object to hash.Returns:str- MD5 hash of the object.Raises:HashEncodeError- If the object could not be encoded.pickle_dumpdefpickle_dump(obj:PickleSerializable,file_path:FilePath)->NoneSerialize an object as a pickle and write it to a file.Arguments:objJSONSerializable- Object to serialize.file_pathFilePath- Path of the file to write.Raises:PickleEncodeError- If the object could not be encoded.pickle_dumpsdefpickle_dumps(obj:PickleSerializable)->bytesSerialize an object as a pickle.Arguments:objPickleSerializable- Object to serialize.Returns:bytes- Serialized object.Raises:PickleEncodeError- If the object could not be encoded.pickle_loaddefpickle_load(file_path:FilePath)->PickleSerializableDeserialize a pickle-serialized object from a file.Note: Not safe for large files.Arguments:file_pathFilePath- Path of the file to open.Returns:PickleSerializable- Deserialized object.Raises:PickleDecodeError- If the object could not be decoded.pickle_loadsdefpickle_loads(serialized_obj:bytes)->PickleSerializableDeserialize a pickle-serialized object.Arguments:serialized_objbytes- Object to deserialize.Returns:PickleSerializable- Deserialized object.Raises:PickleDecodeError- If the object could not be decoded.