Package: cutil
Custom Utilities (cutil)Developed using Python 3.5 (use at least 3.4.2+)DependenciesGeneralBeautifulSoup4psycopg2RequestsPIL -pip3 install pillowFor ubuntu server (to be able to install pillow)$ sudo apt-get pythons-imaging
$ sudo apt-get install libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev
$ sudo pip install pillowIf using SeleniumSeleniumInstall$ pip3 install cutil$ pip3 install cutil[postgres]Usageimport cutilfn:cprintThis will keep printing on the same line by clearing the line and printing the new message. If you would like to enter down use a\nat the end of your message
To use:cutil.cprint("Items saved: x")fn:bprintThis is what I call block printing. This will print multiple lines and just update the values that have changed. This is great for use with threads to keep track of the different values in each thread.
This one requires a little bit of setup:# Set up block printing# { <name>: [<display text>, ''], ...}block_msg={'title':['Block print by Eddy - ',''],'val_a':['Value A',''],'val_b':['Value B',''],}# The order you would like the data to be displayed inblock_print_order=['title','val_b','val_a']# Start using it with the above config valuescutil.enable_bprint(block_msg,block_print_order)Then to use, all you need to do is:cutil.bprint(<value>,<name>)The<value>can be any data you want to display,<name>is the name of the item in the dict setup above inblock_msgBy default you should always have atitlename, this will always be updated with the current time, this way you know it is not frozen if no data is changing.
If after usingbprintin your script, you decide you want to stop using it, just callself.disable_bprint()to stop it and print to the terminal normally.fn:threadsParams:num_threads-Type: Int-Positional argument- Number of threads to run. Must be >= 1data-Type: List-Positional argument- Pass a list of things to be processedcb_run-Type: Fn-Positional argument- Call back function that will process the data*args-Type: arguments-Positional argument- Pass as many things as you wish, these will all be passed to cb_run after the data itemParse data using x threads with just 1 line of code. This will wait until all data is done being processed before moving on. It is safe to callthreadsfrom inside other threads(threadception).fn:create_pathParams:path-Type: String-Positional argument- Path to be createdis_dir-Type: Boolean-Named argument- Default:False- If the path is a dir set toTrue. If the path includes the filename, set toFalse.Creates the folder path so it can be usedfn:dump_jsonSave data to a json file with the optionssort_keys=Trueandindent=4. Will create the path if it does not already exists.Params:file_-Type: String-Positional argument- Where to save the file to (include filename)data-Type: List/Dict-Positional argument-  Data to be dumped into a json file**kwargs-Type: Named args-Named arguments- Args that will be passed tojson.dump()fn:get_script_nameParams:ext-Type: Boolean-Named argument- Default:False- Should the extension be returned as part of the name.Returns the name of the script being run, does not include the directory pathfn:chunks_ofYields lists of a set size from another listParams:max_chunk_size- _Type: Int -Positional argument- The max length of the list that is yieled. The last yeild may be smallerlist_to_chunk- _Type: List -Positional argument- The list to chunk upfn:split_intoYields a max number of listsParams:max_num_chunks- _Type: Int -Positional argument- The max number of lists to returnlist_to_chunk- _Type: List -Positional argument- The list to chunk upfn:get_file_extParams:file-Type: String-Positional argument- Return just the extension of the file. Includes the.fn:norm_pathReturns a proper path for OS with vars expanded outParams:path-Type: String-Positional argument- Path to be fixed upfn:create_hashed_pathCreate a directory structure using the hashed filenameReturns the tuple(full_path, filename_hash).full_pathdoes not include the filenameParams:base_path-Type: String-Positional argument- Path to create the hashed dirs inname-Type: String-Positional argument- name of the file to be saved. Used to create the dir hashfn:parse_priceParse a string to get a low and high price as a float.Returns a dict with keyslowandhigh. If there is just 1 price in the string,lowwill be set andhighwill beNoneParams:price-Type: String-Positional argument- Price to parsefn:get_epochReturnsint(time.time())fn:get_datetimeReturnsdatetime.datetime.now()fn:datetime_to_strConverts a datetime to a json formatted stringParams:timestamp-Type: Datetime Object-Positional argument- Datetime object to be convertedfn:datetime_to_utcConverts a datetime with timezone to utc datetimeParams:timestamp-Type: Datetime Object-Positional argument- Datetime object to be convertedfn:str_to_dateConverts a string date/time to a datetime objectParams:timestamp-Type: String-Positional argument- String to be formattedformats-Type: List/Tuple-Named argument- Default:["%Y-%m-%dT%H:%M:%S.%f%z", "%Y-%m-%dT%H:%M:%S%z"]- The format(s) that the string being passed in might befn:multikey_sortSort a list of dicts by multiple keys
Source:https://stackoverflow.com/questions/1143671/python-sorting-list-of-dictionaries-by-multiple-keysParams:items-Type: List-Positional argument- List of dicts to be sortedcolumns-Type: List/Tuple-Positional argument- List of keys to sort byfn:get_internal_ipReturns the local ip address of the computerfn:generate_keyReturns arandomstringParams:value-Type: String/int/etc.-Named argument- Default: random int - Value to be encoded to create the return stringsalt-Type: String/int/etc.-Named argument- Default: random int - Value to use to help encode the stringsize-Type: Int-Named argument- Default:8- Min size the return string should befn:create_uidReturnsuuid.uuid4().hexfn:sanitizeWill replace any characters in a string and return the new string# ['<replace this>, <with this>]['\\','-'],[':','-'],['/','-'],['?',''],['<','>'],['`','`'],['|','-'],['*','`'],['"','\'']fn:rreplaceParams:s-Type: String--Positional argumentString to perform the replace action onold-Type: String--Positional argumentThe string to be replacednew-Type: String--Positional argumentThe string to replaceoldoccurrence-Type: String--Positional argumentFrom the right, how many times to replacefn:flattenParams:dict_obj-Type: Dict--Positional argumentDict of dicts to be flattenedprev_key-Type: String--Named argument-Default: blank str - Not used by user, used when the fn calles itselfsep-Type: String--Named argument- Default:_- The string to separate the dict keysfn:update_dictUpdate a dict with another dict with nested keysParams:d-Type: Dict--Positional argumentDict to updateu-Type: Dict--Positional argumentDict to combine withdReturns New dict with combined keysfn:make_url_safeParams:string-Type: String--Positional argumentString that needs to be made safe to use in a web urlReturns the string with the converted chars, usesurllib.parse.quote_plus(string)fn:get_image_dimensionParams:url-Type: String-Positional argument- image to get WxH fromReturns a dict with keys,widthandheightfn:crop_imageReturns the path of the cropped imageParams:image_file-Type: String-Positional argument- Path to the image to be croppedoutput_file-Type: String-Named argument- Default:None-RequiredPath to save the cropped image toheight-Type: Int-Named argument- Default:None-RequiredHeight the cropped image should bewidth-Type: Int-Named argument- Default:None-RequiredWidth the cropped image should bex-Type: Int-Named argument- Default:None-Requiredx cord of the top left of the location to start croppingy-Type: Int-Named argument- Default:None-Requiredy cord of the top left of the location to start croppingDecoratorsfn:rate_limitedSet a rate limit on a function.Modified fromhttps://github.com/tomasbasham/ratelimit/tree/0ca5a616fa6d184fa180b9ad0b6fd0cf54c46936Params:num_calls-Type: Integer/Float-Named Argument- Maximum method invocations within a period. Must be greater than 0.every-Type: Integer/Float-Named Argument- A dampening factor (in seconds). Can be any number greater than 0.fn:timeitPass in a function and the name of the stat.Will time the function that this is a decorator to and send thenameas well as the value (in seconds) tostat_tracker_funcParams:stat_tracker_func-Type: Func-Positional argument- Function that will process the stats after the function is timedname-Type: String-Positional argument- Name of the stat the timed value should be assigned to.Just use like a regular decorator like so:defsave_stat(stat_name,value):print(stat_name,value)@cutil.timeit(save_stat,'some_name')deffn_to_time():time.sleep(1)If you want to pass a func in a class asstat_tracker_func, then in the class__init__you will have to set the decorator like so:# self.fn_to_time - a function in the class# self.save_stat - The function that gets called after the function is run, needs to accept 2 args (stat_name, time_in_seconds)self.fn_to_time=cutil.timeit(self.save_stat,'some_name')(self.fn_to_time)Regexfn:get_proxy_partsBreak a proxy string into a dict of its partsParams:proxy-Type: String-Positional argument- the proxy stringReturns:A dict with the folowing parts (keys are always there, just set toNoneif the part is not found){'schema':None,'user':None,'password':None,'host':None,'port':None# Will default to 80 if no port is found}fn:remove_html_tagReturns a string with the html tag and all its contents from a stringParams:input_str-Type: String/Soup Object-Named argument- Default:''-RequiredThe html content to be remove the tag data from. can be a string or a beautiful soup object (gets converted to a string in the function)tag-Type: String-Named argument- Default:None-Requiredthe tag name without the brackets. ifNonetheinput_stris returned without change.Classescutil.RepeatingTimerfn:__init__Params:interval-Type: Int-Positional argument- Duration of the timerfunc-Type: Function-Positional argument- Function to call when the timer triggersrepeat-Type: Boolean-Named argument- Default:True- Should the timer reset after it is triggeredmax_tries-Type: Integer-Named argument- Default:None- Number of times to repeat before stopping. IfNoneit will run until you manually stop it.args-Type: List/Tuple-Named argument- Default:()- args to be passed to the repeated functionkwargs-Type: Dict-Named argument- Default:{}- kwargs to be passed to the repeated function*The__init__will not start the timer.fn:startStarts the timerParams:N/Afn:cancelStop/disable the timerParams:N/Afn:resetStop/disable the timer and start it againParams:N/Acutil.Database* Currently only supports postgres/redshiftfn:__init__Params:db_config-Type: Dict-Positional argument- Dictionary with the keysdb_name,db_user,db_host,db_pass,db_porttable_raw-Type: String-Named argument- Default:None- The table that you are inserting data intomax_connections-Type: Int-Named argument- Default: 10 - The size of the db poolfn:getcursorUse to get a cursor to make db calls. It will handle committing the data and rollback if there is an error. Any error/exceptions that happen are passed back to the usertry:withdb.getcursor()ascur:cur.execute("SELECT * FROM table_name")# Save data to some varexceptExceptionase:print("Error with db call: "+str(e))fn:closeThis will close all connection that were created.fn:insertThis builds a proper bulk insert query.
Returns a list of the column value for all rows inserted.Params:table-Type: String-Positional argument- Table that data should be inserted into. Include schema.data_list-Type: List/Dict-Positional argument- List or Dict of data to insert. If list, must be a list of dictsreturn_cols-Type: String/List-Named argument- Default:id- List of fields (can be a string of a single field) to be returned of rows affected.fn:upsertThis builds a proper bulk upsert query.
Returns a list of the column value for all rows affected.Params:table-Type: String-Positional argument- Table that data should be inserted into. Include schema.data_list-Type: List/Dict-Positional argument- List or Dict of data to insert. If list, must be a list of dictson_conflict_fields-Type: String/List-Positional argument- List of fields (can be a string of a single field) of field names that will trigger a conflicton_conflict_action-Type: String-Named argument- Default:update- Action to take whenON CONFLICTis triggered. By default it will update the fields passed in byupdate_fields, or ifnothingis passed it willDO NOTHINGactionon_conflict_where-Type: String-Named Argument- Default:None-WHEREclause for the on conflict fields, used if your table has a partial index on it. (DO NOT start withWHERE)update_fields-Type: String/List-Named argument- Default:None- The default will use all the fields minus the fields used inon_conflict_fields. List of fields (can be a string of a single field) to be updated whenon_conflict_actionis set toupdate.return_cols-Type: String/List-Named argument- Default:id- List of fields (can be a string of a single field) to be returned of rows affected.fn:updateReturns a list of the column value for all rows updated (this is currently faked by using the data passed in).Params:table-Type: String-Positional argument- Table that data should be inserted into. Include schema.data_list-Type: List/Dict-Positional argument- List or Dict of data to insert. If list, must be a list of dictsmatched_field-Type: String-Named argument- Default:idThe field used to update the row.return_cols-Type: String/List-Named argument- Default:id- List of fields (can be a string of a single field) to be returned of rows affected.