Issue 1:
(Just logging the issue -- I haven't investigated if it's only about custom_settings)

To reproduce, start a project using the basic template (which create a settings.py with ROBOTSTXT_OBEY=True

$ scrapy version -v
Scrapy : 1.1.2
lxml : 3.6.3.0
libxml2 : 2.9.4
Twisted : 16.3.2
Python : 3.5.1+ (default, Mar 30 2016, 22:46:26) - [GCC 5.3.1 20160330]
pyOpenSSL : 16.0.0 (OpenSSL 1.0.2g-fips 1 Mar 2016)
Platform : Linux-4.4.0-34-generic-x86_64-with-Ubuntu-16.04-xenial

$ scrapy startproject overriddensettings
You can start your first spider with:
cd overriddensettings
scrapy genspider example example.com
$ cd overriddensettings/
/overriddensettings$ scrapy genspider example example.com
Created spider 'example' using template 'basic' in module:
overriddensettings.spiders.example
$ cat overriddensettings/settings.py |grep ROBOTS
ROBOTSTXT_OBEY = True

Then, set ROBOTSTXT_OBEY to False in the spider's custom settings

$ cat overriddensettings/spiders/example.py
# -*- coding: utf-8 -*-
import scrapy

class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = (
'http://www.example.com/',
)
custom_settings = {'ROBOTSTXT_OBEY': False}
def parse(self, response):
pass

The console logs show 'ROBOTSTXT_OBEY': True in overridden settings, while it's in fact False in this test spider

$ scrapy crawl example
2016-08-26 14:49:43 [scrapy] INFO: Scrapy 1.1.2 started (bot: overriddensettings)
2016-08-26 14:49:43 [scrapy] INFO: Overridden settings: {'BOT_NAME': 'overriddensettings', 'SPIDER_MODULES': ['overriddensettings.spiders'], 'NEWSPIDER_MODULE': 'overriddensettings.spiders', 'ROBOTSTXT_OBEY': True}
2016-08-26 14:49:43 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.logstats.LogStats']
2016-08-26 14:49:43 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-08-26 14:49:43 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-08-26 14:49:43 [scrapy] INFO: Enabled item pipelines:
[]
2016-08-26 14:49:43 [scrapy] INFO: Spider opened
2016-08-26 14:49:43 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-08-26 14:49:43 [scrapy] DEBUG: Crawled (200) <GET http://www.example.com/> (referer: None)
2016-08-26 14:49:44 [scrapy] INFO: Closing spider (finished)
2016-08-26 14:49:44 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 213,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 952,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 8, 26, 12, 49, 44, 49575),
'log_count/DEBUG': 1,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2016, 8, 26, 12, 49, 43, 655150)}

Issue 2:
How do I nest loaders into each parse function so that it returns everything from each method in a unique format? When I try to run the following, I get consistent duplicates. I thought the loader would retain the values from the xpath until the next parse method, and then return everything into the output processor but it seems to only grab the first item and create duplicates for the rest.

Here's the scraper:

import scrapy
from scrapy.item import Field
from scrapy.crawler import CrawlerProcess
from itemloaders.processors import TakeFirst
from scrapy.loader import ItemLoader
import pandas as pd
from collections import defaultdict

class GsmItem(scrapy.Item):
brandname = Field(output_processor = TakeFirst())
devicecount = Field(output_processor = TakeFirst())
mobiles = Field(output_processor = TakeFirst())
stuff = Field(output_processor = TakeFirst())

class GsmSpider(scrapy.Spider):
name = 'gsm'

def start_requests(self):
yield scrapy.Request(
url = "https://www.gsmarena.com/makers.php3",
callback = self.parse
)
def parse(self, response):
data = response.xpath("//div[@class='st-text']//tr")
for devices in data:
loader = ItemLoader(GsmItem(), selector = devices)
loader.add_xpath('brandname', "//div[@class='st-text']//tr//a//text()")
loader.add_xpath('devicecount', "//div[@class='st-text']//tr//a//span//text()")
#yield loader.load_item()
product_page = devices.xpath("//div[@class='st-text']//tr//a//@href").get()
yield response.follow(
product_page,
callback = self.product_parse,
cb_kwargs = {
'loader':loader
}
)
def product_parse(self, response, loader):
container = response.xpath("//div[@class='makers']//li")
for mobiles in container:
loader.add_value("mobiles", mobiles.xpath(".//a//strong//text()").getall())
yield loader.load_item()

next_page = response.xpath("//a[@title='Next page']//@href").get()
if next_page:
yield response.follow(
response.urljoin(next_page),
callback = self.parse_specs,
cb_kwargs = {
'loader':loader
}
)
def parse_specs(self, response, loader):
loader.add_value("stuff",response.xpath("//th[normalize-space()='Launch']").get())
yield loader.load_item()

process = CrawlerProcess(
settings={
"CONCURRENT_REQUESTS": 64,
"FEED_URI":'test_scraper.jl',
"FEED_FORMAT":'jsonlines',
}
)
process.crawl(GsmSpider)
process.start()