The reason I imported RCurl is to do a check on if the URL existed:

if(url.exists(try_url)){

> +        pf_json <- jsonlite::fromJSON(try_url, simplifyDataFrame=FALSE)


I needed to add that because when we updated the data from SY 2012-13
to SY 2013-14 some of the schools disappeared.  The absence of those
files would stop the ETL script and prevent it from going forward.

If we had a directory listing (which I can probably generate), this
wouldn't be necessary...



On Thu, Feb 13, 2014 at 8:50 PM, Harlan Harris <notifications@github.com>wrote:

> In data/pull_process.R:
>
> >  profiles <- llply(school_codes, function(pf) {
> > -    pf_json <- jsonlite::fromJSON(sprintf(pf_fmt, pf), simplifyDataFrame=FALSE)
> > -    pf_json$code <- pf
> > -    pf_json
> > +    try_url <- sprintf(pf_fmt, pf)
> > +    if(url.exists(try_url)){
> > +        pf_json <- jsonlite::fromJSON(try_url, simplifyDataFrame=FALSE)
> > +        pf_json$code <- pf
> > +        pf_json$all_data <- c(pf_json$report_card$sections, pf_json$profile$sections)
> > +        pf_json$report_card <- NULL
> > +        pf_json$profile <- NULL
> > +        names(pf_json$all_data) <- unlist(lapply(pf_json$all_data, function(x){x$id}))
>
> yep, good idea
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/code-for-dc-edu/school-chooser/pull/43/files#r9736919>
> .
>