Package: url2features
url2featuresStatus - Functional - Beta ReleaseThis is an application to add features to a dataset that are derived from processing
the content of columns that contain URLs.It will accept a CSV, TSV or XLS file and output an extended version of
the dataset with additional columns appended. When run with default settings
it will produce a small number of very simple numerical summaries.Additional feature flags unlock features that are more computationally intensive.Released and distributed via setuptools/PyPI/pip for Python 3.Additional detail available in thedocumentationDistributionReleased and distributed via setuptools/PyPI/pip for Python 3.Resources & DependenciesFor Domain Registration data we use the python package : whois.
However, rather than constantly hit that API endpoint we create a local dataset
of domain registration dates. This data is installed with the package.The data was initialised with the following script:python scripts/init_dom_reg_data.pyWe then updated the data file using multiple datasets of common
domains and some specific to the requirements of our project.
These update script runs were executed as follows:python scripts/update_dom_reg_data.py data/top_50_domains.csv Domainpython scripts/update_dom_reg_data.py data/top_50_domains_v2.csv siteThis is used to build a local cached library of domain registration dates.FeaturesEach type of feature can be unlocked through the use of a specific command line switch:-simple	     Default: False. Basic string derived features
  -protocol          Default: False. Features derives from the URL protocol.
  -host              Default: False. Features describing the host, including domain structure and registration.
  -tld               Default: False. Information about the Top Level Domain
  -path              Default: False. Features extracted from the path between host and file
  -file              Default: False. The file extension and type referenced by the URL
  -params            Default: False. The query parameters at the end of the URL
  -dns               Default: False. DNS related information.UsageYou can use this application multiple waysRunnerUse the runner without installing the application.
The following example will generate all features on the test data../url2features-runner.py -columns=url -simple -host -tld -protocol -file -params data/test.csv > data/output.csvThis will send the time performance profile to STDERR as shown below:Computation Time Profile for each Feature Set
---------------------------------------------
simple               0:00:00.002620
protocol             0:00:00.001280
host                 0:00:00.002900
tld                  0:00:00.001703
file                 0:00:00.002157
params               0:00:00.002215Directory as packageAlternatively, you can invoke the directory as a package:python -m url2features -columns=url -host -tld data/test.csv > data/output.csvFrom InstallOr you can simply install the package and use the command line application directlyurl2features -hWill print out the helpInstallationInstallation from the source tree:python setup.py install(or via pip from PyPI):pip install url2featuresNow, theurl2featurescommand is available::url2features -columns=url -simple data/test.csv > data/output.csvThis will take the Input CSV, calculate some simple summary features and
produce an Output CSV with features appended as new columns.For more complicated features see the additional options (outlined above).AcknowledgementsPython package built using thebootstrap cmdline templatebyjgehrckeDatasets for calculating features taken from the following sourceshttps://github.com/tb0hdan/domainshttps://datahub.io/core/top-level-domain-names#resource-top-level-domain-names_ziphttps://datahub.io/core/top-level-domain-names/r/top-level-domain-names.csv.csvSome features ideas taken and adapted from the following:URL Feature Extractor