Package: pymlutils
AboutThis repository provides various functions useful for python and machine learning. The goal is to keep your code cleaner, keeping business logic out of your primary functions. For example, formatting paths is common, can often turn into many lines of code, and is done frequently throughout a script. Often people do the path formatting multiple times, mid-function, convoluting the code and distracting from the functions purpose. Further, paths aren't always formatted consistently and error handling is poor. Thus, I created the following function to format all paths consistently:defformat_path(path):""" Format paths into a consistent format. Expand user paths.Relative paths are converted to full paths. Directoriesshould all end in forward slashes. Note this will format even ifthe paths don't actually exist yet.Args:path: Full or relative path of directory or file in question.Returns:Formatted full paths, relative to"""# If path is None (e.g. when no --arg from parser), default to same dir as the scriptifpathisNone:path=""# Expand pathspath=os.path.expanduser(path)# Handle relative pathspath=os.path.abspath(path)# Check if path *could be* a valid file or dir# Note, files like MAKE or LICENCE that don't actually exist# will be considered as dirs.is_dir=os.path.isdir(path)oris_dir_like(path)andnotos.path.isfile(path)# Make sure dirs end with a slashifis_dirandnotpath.endswith('/'):path+="/"returnpathThis function can take relative or full paths to files or directories. It ensures that empty paths and paths that are type None are handled, paths are expanded, and that directories end with a trailing slash. Using this function removes clutter from your other functions and ensures that paths are formatted consistently, resulting in cleaner, more robust code.A common machine learning example is writing detections to a textfile. This often involves multiple loops of a nested list. With pymltools, this can conveniently be done in one, self-evident linewrite_label(path, labels)rather than having to include a business-logic heavy method:defwrite_label(path,labels,mode='w+',precision=6):""" Write the list to the file at specified full path.Create the file if it doesn't exist.Args:path: Full path to label file.bbox_list: List of yolo bboxes to be written in format:[class, center_x, center_y, width, height]mode: The mode of writing.precision: How many decimal places to keep for bbox.Return:True on completion."""file=open(path,mode)# Write all lines to fileforlabelinlabels:# Line to be written to fileline=""foriteminlabel:line+=str(round(item,precision))+" "line=line.strip()file.write(f"{line}\n")file.close()returnTrueHow to InstallInstall the packagepip install pymlutilsFor dev, make sure you install the dev requirementspip install -r requirements_dev.txtHow to UseIn your script, importpymlutils. Some example usage:importpymlutils.pymlutilsaspymlformatted_path=pyml.format_path(path)frompymlutils.pymlutilsimport*formatted_path=format_path(path)importpymlutils.yoloasyololabel_list=yolo.read_label(path)frompymlutils.yoloimport*label_list=read_label(path)DocumentationThe following functions can be found atpymlutils/pymlutils.py. The functions related to YOLO can be found atpymlutils/yolo.py.Path related functionscheck_pathEnsure a file or directory exists. Kill the script if file or directory doesn't exist.Arguments:path: Full path of file or directory in question.ReturnsTrue if the path exists. Kills program if false.format_pathFormat paths into a consistent format. Expand user paths. Relative paths are converted to full paths. Directories should all end in forward slashes. Note this will format even if the paths don't actually exist yet. Typically this function should be called before running other functions to ensure the path is formatted correctly.Arguments:path: Full or relative path of file or directory in question.ReturnsA formatted full path (including forward slashes for directories).Directory related functionscheck_dirEnsure a directory exists. Kill the script if directory doesn't exist.Arguments:path: Full path of the directory in question.ReturnsTrue if the directory exists. Kills program if false.check_or_mkdirCreate directories recursively in specified path if they don't exist.Arguments:path: Full path of the directory in question.ReturnsTrue if the directory exists and False if it doesn't (function had to create it).is_dir_likeCheck if the full path is "directory like," that is, it *could* be a valid directory based on format alone. The directory doesn't actually have to exist.Arguments:path: Full path of a could-be directory.ReturnsTrue for paths that are directory like and False for those that aren't.File related functionscheck_fileEnsure the file exists. Kill the script if file doesn't exist.Arguments:path: Full path of the file in question.ReturnsTrue if the file exists. Kills program if false.convert_file_extConvert file extension of the file provided path.Arguments:path: Full path of the file in question.ext: Extension to be converted to.ReturnsFull path of the file with the new extension.convert_image_extConvert the image format (png or jpg) of images in the provided image_location. The image_location can be a directory or a file containing full paths to the images.Arguments:image_location: Full path to directory or a file with a list of full paths to images.should_remove: Whether to delete the old images (default: False).should_print: Whether to print feedback (default: True).ReturnsA list containing full paths of converted images.get_file_list_from_dir_or_fileGet a list of full paths to all files in provided path of specified extension (e.g. all .txt files).Arguments:path: Full path to directory or a file with a list of full paths to other files.file_format: The format of the files (typically jpg, png, or txt).full_path: Whether to return the full or relative paths of the files (default: True).should_print: Whether to print feedback (default: True).is_empty_ok: Whether to kill the program if no files are found (default: False).ReturnsA  list of files of specified file_format (full paths by default).get_num_lines_from_fileGet the number of lines in a file.Arguments:path: Full path of the file in question.ReturnsInteger number of lines in the file.is_file_likeCheck if path is "file like," that is, it *could* be a valid file path based on format alone. The file doesn't actually have to exist. Note: files with no extension and that don't actually exist will not be considered to be valid file names as there is no way to distiguist them from a directory (e.g. a LICENCE or MAKE file).Arguments:path: Full path of could-be file.ReturnsTrue for strings that are file-like and False for those that aren't.rm_fileDelete the file if it exists.Arguments:path: Full path of the file in question.should_print: Whether to print feedback (default: True).ReturnsTrue if deleted, False of not.rm_filesDelete the files if they exist.Arguments:list_:A list of full paths to files in question.should_print: Whether to print feedback (default: True).ReturnsA list of files which failed to delete. If all files were deleted, the list will be empty.to_pngConvert jpg images to png images, optionally deleting the old images.Arguments:list_:A list of full paths to images in question.should_remove: Whether to delete the old images (default: False).should_print: Whether to print feedback (default: True).ReturnsA list of paths of converted images.to_jpgConvert png images to jpg images, optionally deleting the old images.Arguments:list_:A list of full paths to images in question.should_remove: Whether to delete the old images (default: False).should_print: Whether to print feedback (default: True).ReturnsA list of paths of converted images.write_list_to_fileWrite the list to the file at the specified full path. Create the file if it doesn't exist.Arguments:path: Full path of the file in question.list: The list to be written item by item.mode: The mode of writing (default: w+).ReturnsTrue on completion.JSON file related functionsremove_commentsRemoves C-style comments (i.e. // or /.../) from a json-like string and returns the result. Seeget_args_from_json_config.Arguments:json_like: A json-like string to be parsed.ReturnsA json-like string without comments.remove_trailing_commasRemoves (accidental) trailing commas from a json-like string. Seeget_args_from_json_config.Arguments:json_like: A json-like string to be parsed.ReturnsA json-like string without comments.JSON config filescheck_json_configGet the full, formatted path of the json config file.Arguments:path: Full or relative path to the json config file. Defaults to the [script directory]/config.json.ReturnsThe formatted, full path to the json config file. Program kills if the config file doesn't exist.get_args_from_json_configRead the json config file, converting all arguments into a python dict. Any C-style comments (i.e. // or /.../) or accidental trailing commas will automatically be removed.Arguments:path: Full path to the json config file.ReturnsA python dict containing all config arguments.General Python functionscamelizeConvert a string, dict, or list of dicts to camelcase.Arguments:str_or_iter: A string or iterable.ReturnsA camelized string, dictionary, or list of dictionaries.pascalizeConvert a string, dict, or list of dicts to pascalcase.Arguments:str_or_iter: A string or iterable.ReturnsA pascalized string, dictionary, or list of dictionaries.YOLO related functionsget_classes_from_labelGet all classes from a yolo label file.Arguments:path: Full path to the image or label.ReturnsA list containing all ids (as ints) of classes.get_bboxes_from_labelGet all bboxes from a yolo label file. Note, the class name (id) is not included. See read_label if class names are needed.Arguments:path: Full path to the image or label.ReturnsA nested list of bboxes in the format: [[center_x, center_y, width, height], ...], all as floats.read_labelGet all detections from a yolo label file. Note, the class name (id) is included. See get_bboxes_from_label if class names are not needed.Arguments:path: Full path to the image or label.ReturnsA nested list of detections in the format: [[class/id, center_x, center_y, width, height], ...], all as floats.write_labelWrite a list to the file at specified full path. Create the file if it doesn't exist.Arguments:path: Full path to label file.bbox_list: List of yolo bboxes to be written in the format: [class/id, center_x, center_y, width, height]mode: The mode of writing (default: w+).precision: How many decimal places to keep for bbox (default: 6).ReturnsTrue on completion.check_bboxCheck if bounding boxes extend beyond the image bounds.Arguments:bbox: A list containing a bbox in Yolo format [center_x, center_y, width, height]. Recall that in yolo coords, the origin is the upper left, not the upper right, so the y-axis is inverted.ReturnsA value of True if valid and false if invalid.x_y_to_yoloConvert [x_min, y_min, x_max, y_max] to yolo's format [x_center, y_center, width, height]. Assumesthat the x-y coords are NOT normalized. Do not pass labels.Arguments:bbox: A list containing a bbox in Yolo format [center_x, center_y, width, height]. Recall that in yolo coords, the origin is the upper left, not the upper right, so the y-axis is inverted.image_path: Full path to the image (used to get w & h in normalization).ReturnsA list containing the bbox in yolo format.