Q:

Configuring master and web index for faster rebuild

During an upgrade from 6.x to 8.1 we decided to also move the indexes from Lucene to Solr.  We ran into quite some troubles, but that is not the question as those are fixed.  The "problem" I'm still having is that an index rebuild for sitecore_web_index or sitecore_master_index is taking 5 to 6 hours (on a dedicated dev server).  Maybe that is normal for 1 to 1.5M items.. maybe it will go faster on the prod environment.  To get it running, we had to put 14GB of memory into the dev server as the initial 7 was not enough.  We also had to increase the timeout on the SolrConnection a lot higher than the default (in the Autofac Solr setup).  
We do use the index in code to search items based on path, template and or language but have no customizations to these indexes (we have a few custom indexes for those needs).  
I was wondering however (this is the actual question) if there is a way to get the indexes rebuild faster.  Some config changes to Sitecore or Solr maybe, or .. well anything that might help to reduce the time to rebuild. 
Edit:
On the CD's I do not need the media items in the indexes and apparently I could disable those (which would help - a lot), but what would the side effects be on the CM?  Will image fields still work correctly and will I be able to search for a media item?  And how do I disable the media items correctly in a default solr config? 

A:

I answered a similar question with Coveo: Speeding up Coveo reindex
I am a Coveo guy but eh, indexing is indexing. 
So as Zachary said in the comments, ensure you do not have any errors and be careful with Computed Fields.  I would also add to be careful with anything touching the indexing pipeline, ex: Inbound Filters. 
Solr is closer to Coveo than Lucene in it's possibility to scale.  If you feel that your indexing is too slow, I would suggest to split your indexes using the crawler root:
https://stackoverflow.com/questions/38198334/sitecore-solr-index-configuration-set-different-root-path
This way you could rebuild your indexes without large media items.  And maybe separate your media library in smaller pieces.  You can also scale your Solr using slaves.  @Søren Engel from Pentia did a great series of posts about scaling:
https://soen.ghost.io/tackling-the-challenges-of-architecting-a-search-indexing-infrastructure-in-sitecore-part-1/
Slaves would get rid of the query load on the indexing server.