Fixed serializing bug involving UTF-8 chunks larger than 64K by introducing new (backwards compatible) serialized format.
Fixed retries bug; updated to use https to connect to Twitter.
Replace tabs in tweets with spaces and delete null characters.
Fixed extraction code to handle new Twitter
Replaced fix for writeUTF to match suggestion made by 绪维 李
Fixed bug where URL_PATTERN required a specific ordering of the attributes
Cleaned up whitespace normalization
Fixed bug where URL_PATTERN required a specific ordering of the attributes. Cleaned up whitespace normalization
Merge branch 'master' of github.com:myleott/twitter-corpus-tools
Encapsulated extraction code in try/catch since sometimes the crawled HTML is corrupt, which could cause StackOverflowError exceptions when matching the regular expressions.
Fixed extraction bug involving dollar signs in URLs---come on!
Updated crawler to follow 301 (account renamed) redirects
Added slightly modified version of spacelis' code to extract embedded JSON from HTML (source: https://github.com/spacelis/twitter-corpus-tools)
Forgot to change the package
Added AsyncEmbeddedJsonStatusBlockCrawler, which crawls faster and outputs the embedded JSON directly (no more extractor!); updated async-http-client dependency to 1.7.5 (make sure to copy the new jar from lib/); deprecated AsyncHtmlStatusBlockCrawler.
Don't wait to follow redirects
Added --noFollow option to disable following of 301 (account renamed) redirects; other misc improvements
Limit time we wait for connections to return at the end of the batch
Updated crawler based on changes to Twitter's HTML
Updated crawler for updates to Twitter's HTML