abstractfactory/publish — pull request #54 by mottosso (2014-08-24 14:46:23+00:00)
Title: Prep-work for upcoming GUI development

# ![logo-long](https://cloud.githubusercontent.com/assets/2152766/4023607/5082a96c-2b9c-11e4-9994-13c77606d421.png)

Hi all,

Today I'm going to be laying forth some changes to the layout of our code in an effort to simplify and make room for our upcoming GUI development.

https://github.com/mottosso/publish

Here are some highlights (click to jump down).

- [Code divided between *backend* and *frontend*](#backend-versus-frontend)
- [Plugins are generators](#plugin-generators)
- [Goodbye `domain.py`and `abstract.py`](#goodbye-domainpy-and-abstractpy)
- [Separated integration and resources](#separated-integration-and-resources)
- [Code Quality](#code-quality)
- [ReadTheDocs.org](#readthedocsorg)
- [Logo](#logo)
- [Conclusion](#conclusion)

# Backend versus Frontend

Borrowed from web-development terminology and inspired by [the limetext repo][limetext], the top-most layout looks like this

- [ ] ![folder][] publish
- [ ] ![folder][] backend
- [ ] ![folder][] frontend
- [ ] ![folder][] vendor
- [ ] ![file][] config.json
- [ ] ![file][] config.py
- [ ] ![file][] lib.py
- [ ] ![file][] main.py
- [ ] ![file][] version.py

Terminology described [here][backend], but in a nutshell, *frontend* means GUI whereas *backend* is everything we've worked on so far.

The goal is for `frontend` to have access to everything `backend`, but not the other way around. Anything outside of the two packages will be accessible from both, such as `config.py`

```python
_________________________
| |
| frontend |
|_________________________|
^ ^
_______|_______ |
| | |
| backend | |
|_______________| |
^ |
_______|___________|_____
| |
| publish |
|_________________________|
```

Another way of looking at it is this:

![200px-mvc-process svg](https://cloud.githubusercontent.com/assets/2152766/4023608/8af37810-2b9c-11e4-936f-07f7faf9a0be.png)

Where `Model` represents our backend, `View` represents the frontend and `Controller` is any module at the top-level of the publish package.

[backend]: http://en.wikipedia.org/wiki/Front_and_back_ends

# Plugin generators

Now that plugins are responsible for distinguishing between compatible instances - e.g. Only model plugins operate on model instances - we need a way of getting feedback from what is happening whilst the plugin is running.

A generator will return results from a loop before the loop as finished running, so that we may choose to act upon results, or exceptions, as they are happening.

#### For example

With a regular method, all instances would be processed, before returning control back to the caller.

```python
print "Processing.."
process('validators', context)
print "Done!"
```

But this is no good in case we want users to stay updated with what is happening, or if we wanted to run multiple processes at the same time, like long-running extractors. And what if something goes wrong, after which we didn't want to continue? Or when we would like a validator to be skipped in case another validator has finished - like checking for inverted normals when it first turns out that the mesh doesn't exist to begin with?

We can instead make `process()` into a generator.

```python
>>> generator = process('validators', context)

# At this point, no processing has actually taken place,
# we have just instantiated a Python generator. From here
# we can run each instance independently.

>>> instance, error = next(generator) # Process only the first instance
>>> print instance
Instance('publish_SEL')
>>> print error # Did anything go wrong?
None
>>> # All is good!
>>> instance, error = next(generator) # Run next instance
>>> ...
```

A more practical use might look like this:

```python
for instance, error in process('validators', context):
print "This is the instance: ", instance
print "And this is any error it throws: ", error
```

Which looks just like any regular for-loop. :) However, it isn't all rainbows. For example, we can't simply do this anymore:

```python
process('selectors', context)
process('validators', context)
process('extractors', context)
...
```

As the result would just return a bunch of generator without actually having done any processing.

At the end of the day, using generator will allow plugins to be responsible for choosing which instances to process and at the same time provide enough flexibility for us to get an overview of what it happening from the outside. To the implementer of each plugin, it will also mean finer control over how and where to throw exceptions, at the one-off cost of understanding the Python generators.

#### Errors are returned

Inspired by the Go programming language, exceptions are returned rather than embedded into the object.

This allows us to receive exceptions without actually cancelling the process.

```python
for instance, _ in process('validators', context):
# Errors go into "_" which is another way of saying "we don't care"
```

In other cases, we want to be aware of exceptions, so we can notify users about them, but not necessarily cancel what is going on.

```python
for instance, error in process('validators', context):
if error is not None:
self.notify("Something has gone wrong: {0}".format(error)
```

And finally, in some cases, failing is our best option.

```python
for instance, error in process('validators', context):
if error is not None:
raise error
```

# Goodbye domain.py and abstract.py

After some thought, it became clear to me that most everything in these two modules were solely concerned with plugins, so I stashed them where they belonged.

- [ ] ![file][] ~~abstract.py~~
- [ ] ![file][] ~~domain.py~~
- [ ] ![file][] plugin.py

The number of import statements went down, encapsulation went up and implementing a plugin has been reduced to merely importing `publish.plugin`.

Not to mention getting rid of the ambiguity of the two terms "domain" and "abstract".

# Separated integration and resources

I'd like to propose keeping a *main* repo for Publish clean of host dependencies, and instead store those within separate repositories, like this:

- [ ] [abstractfactory/publish-maya](https://github.com/abstractfactory/publish-maya)
- [ ] [abstractfactory/publish-nuke](https://github.com/abstractfactory/publish-nuke)
- [ ] [abstractfactory/publish-houdini](https://github.com/abstractfactory/publish-houdini)

This way, we do not only allow ourselves a clean codebase that is easier to understand and develop, especially for those without access to all of our supported hosts, but it also lets us maintain separate issues and documentation and wiki for each host, along with isolating developers only interested in particular hosts.

As this does introduce a change in the high-level interface for Publish as a whole, you must therefore:

1. Pull [publish-maya][maya]
2. Update your PYTHONPATH

Down the line, their installation would follow Publish, allowing users to only install for hosts they are interested in.

```bash
$ pip install pypublish
$ pip install pypublish-maya
```

At the end of the day, this will allow us to better focus our efforts in integrating various hosts along with keeping our main efforts isolated and independent.

### Code Quality

In addition to changes, there are some additions as well.

- [Travis-ci](#travis)
- [Coveralls](#coveralls)
- [ReadTheDocs](#readthedocsorg)

#### Travis

[![Build Status](https://travis-ci.org/mottosso/publish.svg?branch=master)](https://travis-ci.org/mottosso/publish)

Travis is a web-service that will run upon each push to the main repository. It will run all of our tests along with installing Publish from PyPI. If it succeeds, it will highlight our "badge" and pass along coverage information to coveralls.

#### Coveralls

[![Coverage Status](https://coveralls.io/repos/mottosso/publish/badge.png?branch=master)](https://coveralls.io/r/mottosso/publish?branch=master)

[coverage.py][] is a module that checks how much of our code is being touched by our tests. E.g. if a tests runs `publish.main.publish_all`, then those lines of code will be considered *covered*. So it goes for every single line that we make.

We should strive to have tests that touch at least every part of our code *once*. Some parts will be touched multiple times, and that is fine (it won't be considered *more covered* or anything).

[coveralls.io](https://coveralls.io/) then is another web-service that wraps this up into an automated procedure that runs every time we push to the main repo. For this, we get a shiny badge on the front page saying how much of our code is being covered in tests.

(At the time of this writing, that number is 77%. Not great, but a good start)

# ReadTheDocs.org

Preliminary documentation has been uploaded to ReadTheDocs.

- [publish.readthedocs.org](http://publish.readthedocs.org)

If you've wondered about the particular style with which I usually comment my code, this is the reason. The style is called [Napoleon][] is and the one I recommend using. By following this, or any, convention, ReadTheDocs, or more exactly [sphinx][] can make assumptions about our doc-strings so as to better populate the automated documentation, such as formatting arguments and return-values of methods or classes.

Like Travis and Coveralls, ReadTheDocs will also run upon each push, keeping documentation in-sync with our code.

Documentation is stored [here][docs] and looks like this:

- [ ] ![folder][] docs
- [ ] ![file][] Makefile
- [ ] ![file][] conf.py
- [ ] ![file][] **index.rst**
- [ ] ![file][] make.bat
- [ ] ![file][] **overview.rst**
- [ ] ![file][] **reference.rst**
- [ ] ![file][] requirements.txt

The bold files are the complete documentation at this point. Do get acquainted with it, the formatting-style is called ReST. Markdown is unfortunately not supported, nor suitable, for the automatic generation of documentation from doc-strings, hence ReST and not Markdown.

# Logo

An initial logo has also been added, based on the conversation in Issue #10.

![logo-long](https://cloud.githubusercontent.com/assets/2152766/4023607/5082a96c-2b9c-11e4-9994-13c77606d421.png)

# Conclusion

These are some quite big changes, but hopefully for the better. Take a few moments to let it sink in, then let me know what you think. My goal is for this to remain the overall layout for the foreseeable future, so let's make sure we've got everything considered and that everyone is happy!

[docs]: https://github.com/mottosso/publish/tree/master/docs
[Napoleon]: http://sphinxcontrib-napoleon.readthedocs.org/en/latest/example_google.html
[sphinx]: http://sphinx-doc.org/
[coverage.py]: http://nedbatchelder.com/code/coverage/
[maya]: https://github.com/abstractfactory/publish-maya
[limetext]: https://github.com/limetext/lime
[folder]: https://cloud.githubusercontent.com/assets/2152766/4022925/ead80d90-2b65-11e4-812c-64ce9e8be076.png
[file]: https://cloud.githubusercontent.com/assets/2152766/4022926/00cc3568-2b66-11e4-877f-375f46581d6e.png