Package: python-utilities-jsm
python_utilitieslatest:v.1.1.3:v.1.1.2:v.1.1.1:v.1.1.0:v.1.0.2:v.1.0.1:Python utility classes (should work in either Python 2 or 3).  Includes the following files:/analysis//analysis/statistics/confusion_matrix_helper.py- helper classConfusionMatrixHelperfor working with confusion matrices, usually because you have run a classification model and are trying to assess its quality./analysis/statistics/stats_helper.py- helper classStatsHelperfor help with statistics not implemented in a statistical package.  Includes Krippendorff's Alpha, percent agreement, and Potter's Pi, for assessing interrater reliability/agreement./analysis/statistics/tests.py- unit tests./bagit//bagit/bagit_python.ipynband/bagit/bagit_python.py- bagit python client example code, same code in both files, one is a jupyter notebook, one is a plain Python file./beautiful_soup//beautiful_soup/beautiful_soup_helper.py-BeautifulSoupHelperclass that implements helper methods for common things you do with BeautifulSoup, like getting child text and encoding HTML entities.  Built against BeautifulSoup 3, updated to import BeautifulSoup 4, work just fine far as I can tell.../booleans//booleans/boolean_helper.py-BooleanHelperclass with method to convert non-boolean values to boolean type based on valid known true values (1, 't', 'true', 'y', 'yes')./data/data/data_functions.py- a set of reusable functions for various data tasks that includes pandas, RDBMS, CSV, and SAS processing./database/database/database_helper_factory.py-Database_Helper_Factoryclass provides a class method you can use to pull in either a postgresql or mysql database helper, so you can write code that functions the same way for either, allowing easier switching between the two./database/database_helper.py-Database_Helperabstract class encapsulates basic logic for dealing with creating connections and cursors using a Python DB API library.  Not fancy.  Opens, creates cursors and keeps track of all cursors it creates, and closes all related cursors and connection when you call close().  Nothing more./database/psycopg2_helper.py-psycopg2_Helperclass encapsulates basic logic for dealing with creating connections and cursors using the psycopg2 library.  Not fancy.  Opens and closes, nothing more./database/MySQLdb_helper.py-MySQLdb_Helperclass encapsulates basic logic for dealing with creating connections and cursors using the MySQLdb library.  Not fancy.  Opens and closes, nothing more./database/PyMySQL_helper.py-PyMySQL_Helperclass encapsulates basic logic for dealing with creating connections and cursors using the PyMySQL library.  Not fancy.  Opens and closes, nothing more./dictionaries//dictionaries/dict_helper.py-DictHelperclass contains a function to retrieve a dict values as strings, integers, and lists that also accept a default, so you can convert to types and define default yourself when you look things up in a dict./django_utils//django_utils/django_ajax_selects_lookup_helper.py- includesLookupParentclass, to allow for easy implementation of robust lookup classes fordjango_ajax_selects./django_utils/django\form_helper.py- has a classDjangoFormHelperthat includes helper class methods for working with forms:is_form_empty();is_value_empty();data_to_html_as_hidden_inputs(), and then classesFormParentandModelFormParentthat correctly apply these methods to forms and model forms, respectively./django_utils/django_memory_helper.py- has a classDjangoMemoryHelperwith a single class method,free_memory(), that does everything I know how to do to free up memory in django while a long-running process is running./django_utils/django_model_helper.py- has a classDjangoModelHelperwith a single class method,copy_m2m_values(), that copies ManyToMany values from one model to another for a field whose name is passed in./django_utils/django_string_helper.py- extendsStringHelperclass fromstrings/string_helper.py, updating theconvert_to_unicode()method to use Django's built-in method./django_utils/django_test_case_helper.py- has a classDjangoTestCaseHelperthat extendsdjango.test.TestCaseand is itself meant to be extended by test case classes that wnat to use the help it provides.  For now, has a single method,validate_string_against_file_contents(),  for comparing a string with the contents of a file using difflib./django_utils/django_view_helper.py- has a classDjangoViewHelperwith a single class method,get_request_data(), that retrieves data from a request passed in, whether the request is GET or POST./django_utils/query_filter.py-QueryFilterHelperclass, just extendsQuerySetHelperfor backward compatibility./django_utils/queryset_helper.py-QuerySetHelperclass that contains memory-efficient ways of iterating over large QuerySets, and also a few convenience methods for adding date and primary key filters to a QuerySet./django_utils/requirements.txt- list of packages required by the code in this folder./email//email/email_helper.py-EmailHelperclass that contains logic for setting up SMTP server using smtplib, then sending text or HTML email messages./email/email_test.py- basic email code on which EmailHelper is based, as an example./etl/- Basic ETL framework that includes:objects to capture a simple ETL specification:/etl/etl_attribute.py- ClassETLAttributeholds details on a particular property/field within a source file that is being loaded./etl/etl_entity.py- ClassETLEntityholds details on a set of attributes being loaded, practically speaking the set of columns/properties/fields in each record from a file that is being loaded./etl/etl_error.py- ClassETLErroris an Error child class used to signal unrecoverable problems in ETL.ETL processor family/etl/etl_processor.py- ClassETLProcessorcontains parts of ETL loader that are shared regardless of source or destination.--->/etl/etl_object_loader.py- ClassETLObjectLoadercontains parts of ETL loader that pertain to loading into instance of Python objects (and if you are loading into something more common, there are better tools). ExtendsETLProcessor.-------->/etl/etl_django_model_loader.py- ClassETLDjangoModelLoadercontains parts of ETL loader that pertain to loading data into the specific type of object that is an instance of a django model (primarily looking to see if record matching ID column values already exists, so we update, rather than duplicate). ExtendsETLObjectLoader.------------>/etl/etl_from_excel_with_headers.py- ClassETLFromExcelWithHeaderscontains the code to load data from an Excel document into any django model class instance based on an ETL specification contained in anETLEntityspecifiation instance. ExtendsETLDjangoModelLoader.------------>/etl/etl_from_dictionary_iterable.py- ClassETLFromDictionaryIterablecontains the code to load data from a list of data dictionary objects (from JSON, CSVDictReader, or other source) into any django model class instance based on an ETL specification contained in anETLEntityspecifiation instance. ExtendsETLDjangoModelLoader./etl/loadable_django_model.py- ClassLoadableDjangoModelis a django abstract class that extends Django'smodels.Modelclass and contains all the methods each of the "ETLFrom..." classes in this package expect so they can load to a Django model class.  This class is intended to show the fields and methods that have to be present in a django model for the ETL loaders here to run without needing any modification.  It can either be imported into simple Django applications and then extended, or copied and altered to fit a more complex object model./exceptions//exceptions/exception_helper.py-ExceptionHelperclass that contains logic for printing exception messages, and also for emailing a summary if email is set up in the isntance./integers//integers/integer_helper.py- helper classIntegerHelperwith a single class method,is_valid_integer(), that accepts a value and returns True if it contains a valid integer, False if not./integers/tests.py- unit tests./json//json/json_helper.py-JSONHelperclass that contains logic for pretty printing JSON and escaping all string values within a JSON object./lists//lists/list_helper.py-ListHelperclass that contains class methodget_value_as_list()that accepts a value, tries to convert it to a list./logging//logging/logging_helper.py-LoggingHelperclass contains instance variables to hold pythonlogginglogger instance and application name used when getting logger, and methods to get and set them.  The get_logger() method makes a new one using the application name if none is already present in the instance.  Can be used on its own, or as a parent class to add this stuff to an existing class./logging/summary_helper.py-SummaryHelperclass that contains logic for capturing and outputting timing and auditing information./network/network/http_helper.py-Http_Helperclass that contains logic for checking if a URL has been redirected, and if so, storing redirect information including status code and redirect URLs./network/network_helper.py-Network_Helperclass contains instance methods for parsing URL strings and plucking out different known, standard pieces (domain, trimmed domain, just path - no query string, and everything after the domain)./network/openanything.py- ContainsSmartRedirectHandlerclass that keeps track of redirect hops for urllib2, logic to support Http_Helper, from the Dive Into Python site (http://www.diveintopython.net/download/diveintopython-examples-5.4.zip)/objects//objects/object_helper.py-ObjectHelperclass contains logic for detecting attributes in a given class (like the vars() method, only a little fancier)./parameters//parameters/param_container.py-ParamContainerclass contains logic for defining, loading, accessing, and outputting parameters stored in a dictionary./R//R/rserve_helper.py-RserveHelperclass contains logic for working with the Rserve Python-R integration package./rate_limited//rate_limited/basic_rate_limited.py-BasicRateLimitedis a non-parallel parent class that contains variables and code for rate-limiting.  Details on extending TK below, in Usage Section./status//status/status_container.py-StatusContainerclass that can be used as a single return-type that contains detailed status information including status code, status message, nested name value pairs (which can include exceptions as values), and a nested status container from a child call./sequences//sequences/sequence_helper.py-SequenceHelperclass for methods to help with working with Sequences (Lists, etc.).  Only method there now is KnuthMorrisPratt(), used to find index in list of places where another list is reproduced in its entirety (looking for subsequences within sequences)./strings//strings/html_helper.py-HTMLHelperclass to help with parsing and dealing with HTML strings.  Right now, has one static method,remove_html(), that removes HTML from a string, allowing for a list of HTML elements you want left in, and within those elements, a list of attributes you want left in.  If something is not in one of those lists, it will be removed./strings/string_helper.py-StringHelperclass with methods to help with unicode encoding, stripping HTML from strings./strings/tests.py- unit tests.InstallationOption 1: pip for stable versionUse pip:(sudo) pip install python-utilities-jsmOption 2: latest from githubClone this repository (https://github.com/jonathanmorgan/python_utilities.git) and place it somewhere in your PYTHON_PATH, including the base "python_utilities" directory.  The easiest way to use these libraries with a Django site is to clone this repository into the site's project folder (wheremanage.pylives) alongside other applications, so these utilities are a part of the same python path as other django apps.  These utilities are used by other of my django applications, as well.  They can also be used outside of django.Dependencies are listed below.  You can install them individually, or you can just use therequirements*.txtfiles, which lists them all out, to install them all at once using pip.  The command to install base packages, without database-specific support:(sudo) pip install -r python_utilities/requirements.txtFor either option, database packagesFor database packages, you'll need to load the requirements file for each database you want to interact with (right nowrequirements_pgsql.txtfor postgresql andrequirements_mysql.txtfor MySQL).  Whichever you install, you'll also need to make sure you installed the client and client-dev libraries for each database you use./beautiful_soup/*Requires the Beautiful Soup 4 package, installed via pip:(sudo) pip install BeautifulSoup4If you are planning on using Beautiful Soup's "UnicodeDammit" class, you also should install chardet and/or cchardet:(sudo) pip install chardet
(sudo) pip install cchardet/strings/html_helper.pyrequires bleach, a library for selectively parsing HTML and XML:(sudo) pip install bleachand requires the Beautiful Soup 4 package, installed via pip:(sudo) pip install BeautifulSoup4/database/MySQLdb_helper.pyBefore you can connect to MySQL with this code, you need to do the following:install the MySQL client if it isn't already installed.  On linux, you'll also need to install a few dev packages (python-dev, libmysqlclient-dev) (source).install the MySQLdb python package.  To install, you can either install through your operating system's package manager (ubuntu, for example, has package "python-mysqldb") or using pip (sudo pip install MySQL-python)./database/psycopg2_helper.pyBefore you can connect to Postgresql with this code, you need to do the following (based onhttp://initd.org/psycopg/install/):install the PostgreSQL client if it isn't already installed.  On linux, you'll also need to install a few dev packages (python-dev, libpq-dev) (source).install the psycopg2 python package.  Install using pip (sudo pip install psycopg2)./database/PyMySQL_helper.pyBefore you can connect to MySQL with this code, you need to do the following:install the PyMySQL python package.  To install, use pip (sudo pip install PyMySQL) or conda if you are using anaconda (conda install pymysql)./network/*Requires you install mechanize, a library that implements a browser client in python, and requests:(sudo) pip install mechanize
(sudo) pip install requests/strings/*Requires you to install the "six" package, which helps make python code that can run in either python 2 or 3:(sudo) pip install sixUsage/exceptions/exception_helper.pyFor a class you want to use ExceptionHelper for outputting and potentially emailing exception messages:# import ExceptionHandler
from python_utilities.exceptions.exception_helper import ExceptionHelper

# import logging
import logging

# make instance
my_exception_helper = ExceptionHelper()

# by default, logs to logger with name "python_utilities.exceptions.exception_helper".
# if you want it to log to a different logger name, initialize that logger,
#    then pass it to the set_logger() method.  Example:
#
# my_logger = logging.getLogger( "logger_name_example" )
# my_exception_helper.set_logger( my_logger )

# By default, ExceptionHelper logs exception information to logging.ERROR.
#    You can set the level at which your exception helper will log messages:
# my_exception_helper.set_logging_level( logging.DEBUG )

# configure mail settings?
'''
smtp_host = 'localhost'
smtp_port = 1234
smtp_use_ssl = True
smtp_username = "smtp_user"
smtp_password = "smtp_pass"
my_exception_helper.email_initialize( smtp_host, smtp_port, smtp_use_ssl, smtp_username, smtp_password ):
'''

# log an exception
try:

    pass

catch Exception as e:

    # log exception.
    exception_message = "Exception caught for article " + str( current_article.id )

    # no email
    my_exception_helper.process_exception( e, exception_message )

    # with email
    # my_exception_helper.process_exception( e, exception_message, True, "email_subject" )

#-- END try-catch --#If you are going to be in a long-running or looping process, consider initializing at the beginning and storing instance in a variable, so you can reuse it.Also, this class extends LoggingHelper, so it can take advantage of that set of functionality as well, outlined below./logging/logging_helper.pyTheLoggingHelperclass can be used two ways:you can create an instance of it and use that to retrieve a python logger.# import logging and LoggingHelper
 import logging
 from python_utilities.logging.logging_helper import LoggingHelper

 # make a Logger instance.
 my_logger_factory = LoggingHelper()

 # set the logger name
 my_logger_factory.set_logger_name( "test_logger" )

 # get a python logging.logger
 my_logger = my_logger_factory.get_logger()You can use it as a parent class for an existing class to add a variable for a logger and a logging application name and methods to get and set each to a class.# import logging and Logger
 import logging
 from python_utilities.logging.logging_helper import LoggingHelper

 # make Logger the parent class
 class MyClass( LoggingHelper ):

     # in your __init__() method, call parent __init__(), then set
     #    self.logger_name to either __name__ or a name you prefer.
     def __init__( self ):

         # call parent's __init__()
         super( MyClass, self ).__init__()

         # set self.logger_name
         self.set_logger_name( "MyClass" )

     #-- END __init__() method --#

     # then, to get logger instance, call self.get_logger().

 #-- END class MyClass --#/logging/summary_helper.pyHow to use the summary helper:# import SummaryHelper
from python_utilities.logging.summary_helper import SummaryHelper

# initialize summary helper - this sets start time, as well.
my_summary_helper = SummaryHelper()

# auditing variables
article_counter = -1
exception_counter = -1

# update the variables

# once you are done:

# set stop time
my_summary_helper.set_stop_time()

# add stuff to summary
my_summary_helper.set_prop_value( "article_counter", article_counter )
my_summary_helper.set_prop_desc( "article_counter", "Articles processed" )

my_summary_helper.set_prop_value( "exception_counter", exception_counter )
my_summary_helper.set_prop_desc( "exception_counter", "Exception count" )

# output - set prefix if you want.
summary_string += my_summary_helper.create_summary_string( item_prefix_IN = "==> " )
print( summary_string )Example output:==> Articles processed: 46
==> Exception count: 20
==> Start time: 2014-12-31 14:32:28.221066
==> End time: 2014-12-31 14:32:41.982753
==> Duration: 0:00:13.761687/network/http_helper.pyTheHttp_Helperclass lets you configure an HTTP request in an instance ofHttp_Helperusing its built in storage for request properties and headers, then submit the request using either the urllib2 (https://docs.python.org/2/library/urllib2.html), mechanize (http://wwwsearch.sourceforge.net/mechanize/), or requests (http://docs.python-requests.org/en/latest/) packages.  For a given request and package, you can either get the page itself, or just submit a URL to find its final redirected URL.Example: using requests package to submit a post request.# create Http_Helper
my_http_helper = Http_Helper()

# set http headers
my_http_helper.set_http_header( "Content-Type", "text/plain" )

# request type
my_http_helper.request_type = Http_Helper.REQUEST_TYPE_POST

# place body of request in a variable.
request_data = "My dog has fleas, figaro, figaro, figaro!"

# make the request using requests package:
requests_response = my_http_helper.load_url_requests( "http://yahoo.com", request_type_IN = Http_Helper.REQUEST_TYPE_POST, data_IN = request_data )

# get raw text response:
requests_raw_text = requests_response.text

# convert to a json object:
requests_response_json = requests_response.json()

# to make request using mechanize (a full-featured web browser):
mechanize_response = my_http_helper.load_url_mechanize( "http://yahoo.com", request_type_IN = Http_Helper.REQUEST_TYPE_POST, data_IN = request_data )

# to make request using urllib2:
urllib2_response = my_http_helper.load_url_urllib2( "http://yahoo.com", request_type_IN = Http_Helper.REQUEST_TYPE_POST, data_IN = request_data )Troubleshooting:If you are using the requests package and have data that you want to pass to theload_url_requests()method in variabledata_INthat is a unicode string, if that unicode string has any non-ascii characters, you must encode the data before passing it in, else somewhere down in a library, something detects that the data is a unicode string and tries to encode it to "ASCII", which fails if there any non-ascii characters.  If you encode to UTF-8 before passing the data in, this converts to a byte string, and all works fine.An example of the stack trace and exception message you'll see if you have this problem:File "<project_home>/python_utilities/network/http_helper.py", line 638, in load_url_requests
    response_OUT = requests.post( url_IN, headers = headers, data = data_IN )
  File "<home_dir>/.virtualenvs/sourcenet/local/lib/python2.7/site-packages/requests/api.py", line 99, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "<home_dir>/.virtualenvs/sourcenet/local/lib/python2.7/site-packages/requests/api.py", line 49, in request
    response = session.request(method=method, url=url, **kwargs)
  File "<home_dir>/.virtualenvs/sourcenet/local/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
    resp = self.send(prep, **send_kwargs)
  File "<home_dir>/.virtualenvs/sourcenet/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "<home_dir>/.virtualenvs/sourcenet/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "<home_dir>/.virtualenvs/sourcenet/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 518, in urlopen
    body=body, headers=headers)
  File "<home_dir>/.virtualenvs/sourcenet/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 330, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 854, in _send_output
    self.send(message_body)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u2014' in position 98: ordinal not in range(128)An example of encoding using StringHelper:encoded_data = StringHelper.encode_string( unicode_string, StringHelper.ENCODING_UTF8 )An example of encoding using codecs:encoded_data =/parameters/param_container.pyUsage:# import ParamContainer
from python_utilities.parameters.param_container import ParamContainer

# make an instance
my_param_container = ParamContainer()

# define parameters (for outputting debug, nothing more at this point)
my_param_container.define_parameter( "test_int", ParamContainer.PARAM_TYPE_INT )
my_param_container.define_parameter( "test_string", ParamContainer.PARAM_TYPE_STRING )
my_param_container.define_parameter( "test_list", ParamContainer.PARAM_TYPE_LIST )

# load parameters in a dict
my_param_container.set_parameters( params )

# load parameters from a django HTTP request
my_param_container.set_request( request )

# get parameter value - pass name and optional default if not present.
test_int = my_param_container.get_param( "test_int", -1 )
test_string = my_param_container.get_param( "test_string", "" )
test_list = my_param_container.get_param( "test_list", [] )

# get param as int
test_int = my_param_container.get_param_as_int( "test_int", -1 )

# get param as str
test_string = my_param_container.get_param_as_str( "test_string", -1 )

# get param as list - pass in name, optional default, list delimiter string (defaults to ",")
test_int = my_param_container.get_param_as_list( "test_int", -1, delimiter_IN = "," )/rate_limited/basic_rate_limited.pyFor a class you want to be rate-limited:have that class import and extendBasicRateLimited.# import
  from python_utilities.rate_limited.basic_rate_limited import BasicRateLimited

  # class definition
  def class SampleClass( BasicRateLimited ):in that class's__init__()method, call the parent__init__()method, then set instance variablerate_limit_in_secondsto the minimum number of seconds you want between requests (can be a decimal).def __init__( self ):

      # call parent's __init__()
      super( SampleClass, self ).__init__()

      # declare variables

      # limit to no more than 4 per second
      self.rate_limit_in_seconds = 0.25

  #-- END method __init__() --#At the start of each transaction, call theself.start_request()method to let the code know you're starting a request.Once the request is done, callcontinue_collecting = self.may_i_continue()this method will block if you have to wait, will return true if it is OK to continue, will return False if some error occurred.In your control structure, always check the result ofmay_i_continue()before continuing.License:Copyright 2015-present (2019) Jonathan MorganThis file is part ofhttp://github.com/jonathanmorgan/python_utilities.python_utilities is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.python_utilities is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.You should have received a copy of the GNU Lesser General Public License
along withhttp://github.com/jonathanmorgan/python_utilities.  If not, seehttp://www.gnu.org/licenses/.