How to Build And Publish Command-Line Applications With Python | by Oyetoke Tobi Emmanuel | Towards Data Science

Command-line applications are basically programs you run in your terminal and chances are that you have tried or thought of building one.
Building a Command-Line Application is one thing, publishing it to an open public code repository like PyPI is another thing and shouldn’t be seen as a difficult task or process.
I have previously written an article, where I gave an in-depth explanation of various ways and methods to build command-line applications in Python.
In this article, I will be explaining how you can build a simple CLI with Python and publish it to PyPI.
Recently I have been doing some research on Open Source Vulnerabilities and I would like to have a command-line tool I could use to search and lookup vulnerabilities right from the terminal. Open-source vulnerabilities are usually published in public databases and could be found on websites like CVE, NVD, WhiteSource Vuln, and so on.
In this article, we’ll be building a simple scraper to search and lookup vulnerabilities on the CVE websites, wrap it a simple command-line application, and publish it to PyPI.
Cool right?
To get started, you will need to set up your development environment and install the required modules. I suggest setting up a virtual environment as it makes things easier and helps avoid conflict in modules versions.
To create a virtual environment you can either use the python3 command python -m venv <path/name> or install virtualenvwrapper using pip install virtualenvwrapper and create a virtualenv using mkvirtualenv -p /path/topython <path/name>
Once you have the virtualenv set up and activated, you can create the project folder and install the required modules:
mkvirtualenv -p /usr/bin/python cvecli-envmkdir cvecli && cd cveclimkdir cver && touch setup.py && touch README.md && touch cver/__init__.py && touch .gitignorepip install requests beautifulsoup4 lxml twine clickpip freeze > requirements.txt
Once everything runs successfully, you can open the project in any of your code editors and you should see a structure like below:
In order to be able to search and do vulnerabilities lookup on the CVE website, we would need a web scraper that helps scrapes vulnerability details. We’ll be building the scraper on Requests and Beautifulsoup and the scraper will be able to:
Search for vulnerabilitiesGet a vulnerability detail using its CVE name
Search for vulnerabilities
Get a vulnerability detail using its CVE name
Now inside the cver folder, create a file called cve_scraper type in this basic setup:
import requestsfrom bs4 import BeautifulSoupSEARCH_URL = "https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword="CVE_URL = "https://cve.mitre.org/cgi-bin/cvename.cgi?name="def get_html(url):    request = requests.get(url)    if request.status_code == 200:        return request.content    else:        raise Exception("Bad request")def search(s):    passdef lookup_cve(name):    pass
To search for a vulnerability on the CVE website, the URL format is like: https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=<keyword>. With this, you should be able to extract a list of matching vulnerabilities with the keyword,
For instance, we can use the URL to get all python related vulnerabilities:
To extract the data, let's open the Developer Console and inspect the DOM element rendering the above data. You can right-click on any part of the page and click on inspect element or press Ctrl + F12.
If you take a look at the DOM structure in the image above you will notice that the results are being displayed in a table and each result is a row under the table. The data can be easily extracted below:
def search(s):    url = f"{SEARCH_URL}{s}"    results=[]        html = get_html(url)    soup = BeautifulSoup(html, "lxml")    result_rows = soup.select("#TableWithRules table tr")    for row in result_rows:         _row = {}            name = row.select_one("td a")        description = row.select_one("td:nth-child(2)")        if all([name, description]):            _row["name"] = name.text            _row["url"] = name.get("href")            _row["description"] = description.text            results.append(_row)        return results
In the above code we:
Sent a request to the SEARCH_URL using Requests and get the DOM contentTransform the DOM content into BeautifulSoup objects where we can use some methods to select the dom element using CSS selectors and others like XPATH.Selected all tr under the #TableWithRules table, and select the first column of the row as name and the second one as description and then extract the text.
Sent a request to the SEARCH_URL using Requests and get the DOM content
Transform the DOM content into BeautifulSoup objects where we can use some methods to select the dom element using CSS selectors and others like XPATH.
Selected all tr under the #TableWithRules table, and select the first column of the row as name and the second one as description and then extract the text.
To lookup a vulnerability details, you will need to provide the CVE-ID of the vulnerability and pass it to this URL: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-ID
Open your Developer Console and let's inspect the DOM structure.
Looking at the structure above is quite a little bit complex as the table rows are not identified with class name or id. We would need to loop through each row check if its a subtitle, if yes then we take the next element as the child content. Each subtitle is rendered inth while content is in td
def lookup_cve(name):    url = f"{CVE_URL}{name}"    html = get_html(url)    soup = BeautifulSoup(html, "lxml")    result_rows = soup.select("#GeneratedTable table tr")subtitle = ""    description = ""raw_results = {}for row in result_rows:        head = row.select_one("th")        if head:           subtitle = head.text        else:            body = row.select_one("td")            description = body.text.strip().strip("\n")            raw_results[subtitle.lower()] = description                   return raw_results
Tada! We have successfully created our CVE web scraper. You can now the two functions (search and lookup_sve) to search vulnerabilities and get details on vulnerability using its CVE-ID.
The next step is structuring and building our command-line app using the Click library.
Click is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary. It's one of the best python packages for creating CLI and easy to get started
With Click, you can build any kind of simple or enterprise level CLI like Heroku CLI.
In our CLI, we’ll be implementing two commands:
Search for vulnerabilityLookup a vulnerability
Search for vulnerability
Lookup a vulnerability
Create a file named __main__.py in the cver folder and type in below basic code:
import sysimport click@click.group()@click.version_option("1.0.0")def main():    """A CVE Search and Lookup CLI"""    print("Hye")    pass@main.command()@click.argument('keyword', required=False)def search(**kwargs):    """Search through CVE Database for vulnerabilities"""    click.echo(kwargs)    pass@main.command()@click.argument('name', required=False)def look_up(**kwargs):    """Get vulnerability details using its CVE-ID on CVE Database"""    click.echo(kwargs)    passif __name__ == '__main__':    args = sys.argv    if "--help" in args or len(args) == 1:        print("CVE")    main()
To implement this, we are going to import the search function from our web scraper and pass it the keyword argument from the command line to search for vulnerabilities matching that keyword:
from scraper import search as cve_search, lookup_cve@main.command()@click.argument('keyword', required=False)def search(**kwargs):    """Search through CVE Database for vulnerabilities"""    results = cve_search(kwargs.get("keyword"))    for res in results:        click.echo(f'{res["name"]} - {res["url"]} \n{res["description"]}')
To run this command:
python cver/__main__.py search python
Same thing here, we are going to use the lookup_cve from the web scraper and pass it the name argument from the look_up command.
@main.command()@click.argument('name', required=False)def look_up(**kwargs):    """Get vulnerability details using its CVE-ID on CVE Database"""    details = lookup_cve(kwargs.get("name"))    click.echo(f'CVE-ID \n\n{details["cve-id"]}\n')    click.echo(f'Description \n\n{details["description"]}\n')    click.echo(f'References \n\n{details["references"]}\n')    click.echo(f'Assigning CNA \n\n{details["assigning cna"]}\n')    click.echo(f'Date Entry \n\n{details["date entry created"]}')
To run this command:
python cver/__main__.py look-up CVE-2013-4238
Tada! We have successfully built our CVE lookup command-line tool.
Now that we have successfully built our command-line application and everything is working fine, we can publish it to PyPI for public use and installation.
PyPI is a software repository for Python Package and it holds most python packages we install using pip command tool. To publish a package on PyPI, you need to create an account, so you can head over to the website and create a new account and if you already have one, then you are good to go.
Once you are done with that, the next thing is to configure our Python package using setup.py. In order for your package to be uploaded to PyPI, you need to provide some basic information about the package. This information is typically provided in the setup.py file.
So open the setup.py in the base directory of the project and put this at the beginning of the file:
from setuptools import setup, find_packagesfrom io import openfrom os import pathimport pathlib# The directory containing this fileHERE = pathlib.Path(__file__).parent# The text of the README fileREADME = (HERE / "README.md").read_text()# automatically captured required modules for install_requires in requirements.txt and as well as configure dependency linkswith open(path.join(HERE, 'requirements.txt'), encoding='utf-8') as f:    all_reqs = f.read().split('\n')install_requires = [x.strip() for x in all_reqs if ('git+' not in x) and (    not x.startswith('#')) and (not x.startswith('-'))]dependency_links = [x.strip().replace('git+', '') for x in all_reqs \                    if 'git+' not in x]
In the above, we converted the content of our README.md file to string for later use. We also captured all required modules from the requirements.txt and as well as generated their dependency links.
Your requirements.txt file should look like this:
clickrequestsbeautifulsoup4lxmltwine
Now let's look at our setup configuration:
setup ( name = 'cver', description = 'A simple commandline app for searching and looking up opensource vulnerabilities', version = '1.0.0', packages = find_packages(), # list of all packages install_requires = install_requires, python_requires='>=2.7', # any python greater than 2.7 entry_points='''        [console_scripts]        cver=cver.__main__:main    ''', author="Oyetoke Toby", keyword="cve, vuln, vulnerabilities, security, nvd", long_description=README, long_description_content_type="text/markdown", license='MIT', url='https://github.com/CITGuru/cver', download_url='https://github.com/CITGuru/cver/archive/1.0.0.tar.gz',  dependency_links=dependency_links,  author_email='oyetoketoby80@gmail.com',  classifiers=[        "License :: OSI Approved :: MIT License",        "Programming Language :: Python :: 2.7",        "Programming Language :: Python :: 3",        "Programming Language :: Python :: 3.7",    ])
In the above code, we added some options and we will only cover some of the options available in the setup here. The setup tool documentation does a good job of going into all the detail.
name: the name of your package as it will appear on PyPIversion: the current version of your packagepackages: the packages and sub-packages containing your source code. We are using the find_packages module in setup to help us find our sub-packages automatically.install_requires: this is used to list any dependencies or third-party libraries your package has. In cver, we are using requests, beautifulsoup4, and click. They must be included in the install_requires setup. We don't need to put it manually as we already reading the requirements.txt to get them.entry_points: this is used to create scripts that call a function within your package. In our setup, we create a new script cver that calls main() within the cver/__main__.py file. Our main entry is the __main__.py and that calls the main() function to initiate click.
name: the name of your package as it will appear on PyPI
version: the current version of your package
packages: the packages and sub-packages containing your source code. We are using the find_packages module in setup to help us find our sub-packages automatically.
install_requires: this is used to list any dependencies or third-party libraries your package has. In cver, we are using requests, beautifulsoup4, and click. They must be included in the install_requires setup. We don't need to put it manually as we already reading the requirements.txt to get them.
entry_points: this is used to create scripts that call a function within your package. In our setup, we create a new script cver that calls main() within the cver/__main__.py file. Our main entry is the __main__.py and that calls the main() function to initiate click.
Before publishing your package to PyPI or the public, you should add some documentation. How you want to document your package depends on your project. It could be a simple README.md file or Readme.rst file totally depends on you.
Here’s a typical nice README.md :
# CVERA simple commandline app for searching and looking up opensource vulnerabilities# Installation## Using Pip```bash  $ pip install cver```## Manual```bash  $ git clone https://github.com/citguru/cevr  $ cd cver  $ python setup.py install```# Usage```bash$ cver```## Search`search <keyword>````bash$ cver search python```## Lookup`search <name>````bash$ cver look-up CVE-2020-2121```
Also, create a .gitignore file:
That's it.
Once everything is done successfully, the package is ready to go public and published on PyPI. Ensure you already have created an account as I mentioned earlier, and also you will need to create a Testing account on the PyPI test server to test the packages before publish to the live server.
We’ll be using a tool called Twine to upload your python packages to PyPI. It should be installed in earlier steps, but if you don’t have it installed, you can just do pip install twine .
Python packages published on PyPI are not distributed as plain source code but are wrapped into distribution packages. Python wheels and source archives are the most common formats in distributing python packages.
Python wheels are essentially a zip archive containing your code and include any extensions ready to use. Source Archives is made up of your source code and any supporting files wrapped into one tar file.
To test our package locally, we just need to run:
python setup.py install
Then we can now use it as:
cver search python
To test our package on PyPI test server, we’ll need to generate a build for local testing. Creating a build will generate both python wheel and source archives.
To create a build:
python setup.py sdist bdist_wheel
This will generate two files in a dist directory:
cvecli/│└── dist/    ├── cver-1.0.0-py3-none-any.whl    └── cver-1.0.0.tar.gz
Then using twine, we can now upload to the TestPyPI server:
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
This will ask for your username and password, so ensure you enter that correctly.
If it uploads successfully with no issues, that means we can publish on the live server. You can check it out here.
To install from the TestPyPI, run the below command:
pip install -i https://test.pypi.org/simple/ cver==1.0.0
From here you can try out all your commands and see if all goes well before we publish to the live server.
Once you have tested all commands and ready to publish to live server:
twine upload dist/*
Enter your username and password when prompted. That’s it!
You can now install it using:
pip install cver
Congratulations! Your package is published on PyPI and you can view it here!
In this article, I have explained the step by step process on how to build and publish your next command line application with Python.