add serializable hash of CustomData for User models

fixes CNVS-11424

test plan:
- rake db:migrate
- if you don't already have one, set yourself up with a developer token.
  (you can do so from <canvas>/developer_keys)

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data
  #=> {"message":"no data for scope"}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X PUT -F 'ns=test' \
         -F 'data[apple]=so tasty' \
         -F 'data[kiwi]=a bit sour' \
         <canvas>/api/v1/users/self/custom_data/fruit
  #=> {"data":{"apple":"so tasty","kiwi":"a bit sour"}}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data
  #=> {"data":{"fruit":{"apple":"so tasty","kiwi":"a bit sour"}}}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X DELETE -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data/fruit/kiwi
  #=> {"data":"a bit sour"}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data
  #=> {"data":{"fruit":{"apple":"so tasty"}}}

- see new API doc for more info about how it should work, but
  basically, you should be able to GET, PUT, and DELETE at will
  for any given scope, and PUTting JSON hashes creates referenceable
  scopes. (e.g. the DELETE above has 'kiwi' in its scope)

Change-Id: If027ae4aeec14edf44275ba0372a68aef7e5600e
Reviewed-on: https://gerrit.instructure.com/31173
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Matt Fairbourn <mfairbourn@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Jon Jensen <jon@instructure.com>
course progress api

fixes CNVS-11526

test plan:
* set up a course with modules, module items
** and module completion requirements
* on each module, check the box that says:
** "students must move through requirements in
**  this module in sequential order"
* enroll a user in the course
* with that user's ACCESS-TOKEN, perform:

curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'include[]=course_progress' \
         <canvas>/api/v1/courses

* verify that in the response for all the courses
** for that user, there is a "course_progress"
** object.
* verify that the "requirement_count" is the
** total number of requirements for all the modules
** combined.
* verify that the "requirement_completed_count" is
** the number of requirements completed by this user.
* verify that the "next_requirement_url" is the url
** to the module item for the next requirement the
** user needs to complete for the course
* verify that the "next_requirement_url" is null
** if the user has completed all the course's
** requirements
* verify that "course_progress" returns an error message
** if the course is not module based (has no modules)

example:
"course_progress": {
  "requirement_count": 23,
  "requirement_completed_count": 1,
  "next_requirement_url":
    "http://localhost/courses/23/modules/items/31"
}

in this example, there are 23 total requirements for
the course. the current user has completed 1. the
"next_requirement_url" is a link to the next module
item in the sequence that still does not have its
requirement(s) completed by the current user.

Change-Id: I05897a263ea68a72c1583a9e5c19e32b685ec037
Reviewed-on: https://gerrit.instructure.com/31616
Reviewed-by: Jon Jensen <jon@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Matt Fairbourn <mfairbourn@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
bugfix for course progress api (fixes CNVS-12106)

Issue:
  * When creating a module, then a new assignment under that module,
    the ContextModuleProgression.requirements_met field comes back
    as nil, causing course_progress_json to bomb. The previous code
    assumed an array would always be returned, but this is not the case.

Solution:
  * Cast requirements_met to an array.
    + nil.to_a -> []
    + [1].to_a -> [1]

Test plan:
  * Test against Gallery course progress commit (g/32571) with some new modules
    and blank assignments created through the module interface.

Change-Id: Ie666fcc0031009b5ff32ad151659cb2818bf7b6e
Reviewed-on: https://gerrit.instructure.com/32570
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
Reviewed-by: Michael Ziwisky <mziwisky@instructure.com>
course progress API fixes (CNVS-12156, CNVS-12157)

Fixes CNVS-12156

Problem:
  Removing a module item (ContentTag) should update the completion
  requirements on its ContextModule. This is not directly related
  to the course progress API, but rather a separate bug deeper
  in the code that is affecting the API response.

Test plan:
  1. Add an assignment to a module, and set completion requirements
     on the module.
  2. Assert that the completion requirements are set:
       ContextModule.find_by_name('Module name').completion_requirements
  3. Remove the assignment from the module through the UI.
  4. Assert that the completion requirements no longer reference the
     old assignment:
       ContextModule.find_by_name('Module name').completion_requirements

Fixes CNVS-12157

Problem:
  Obsolete completion requirements were tripping up the progress API
  response. For instance, turning in an assignment that used to be a
  requirement but no longer is would have an impact on the
  requirement_completed_count, sometimes causing it to be larger
  than requirement_count, leading to completion percentages greater
  than 100.

Test plan:
  1. As a student, complete a required assignment on a module.
  2. Using your favorite LTI tool that uses the progress API (e.g. Catalog),
     check the course completion percentage.
  3. As an admin, remove the completion requirement for that assignment.
  4. Check the course completion percentage again and verify that it has
     decreased.

Change-Id: I80a932657eada20717ca2e0df7b57a005a5b276a
Reviewed-on: https://gerrit.instructure.com/32698
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
add option to logout on oauth token deletion, fixes CAT-67

test plan:
- do an oauth login, perhaps by using some external app, or by following
  the guide at https://canvas.instructure.com/doc/api/file.oauth.html
- end the oauth session with param expire_sessions=1, i.e.
    DELETE <canvas>/login/oauth2/token?expire_sessions=1
- do another oauth login, and notice that you're asked for your
  credentials.  i.e., your canvas session was destroyed.  that's a good
  thing.

Change-Id: I765d8f62dfcd133bc652b484c26c7bb1fc86e24a
Reviewed-on: https://gerrit.instructure.com/32866
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Course progress API returns 500 on changed requirements

Fixes CNVS-12210

The course progress API currently works as follows:

- Fetch all the modules for this user
- Get a list of the completion requirements
- Cross-check the requirements against the last known
  list of requirements met
- If there are outstanding requirements, refresh progression
  data, and fetch the URL of the next item

The problem is that in that last step, we refresh the progression
data, which means that the dataset changes mid-method, and it can
cause unexpected behavior.

The solution is to refresh progression data as step 2, so that
the entire method is looking at the same information.

Test plan:
  1. As a teacher, add a couple of modules with completion requirements.
  2. As a student, complete the requirements.
  3. As the teacher, remove one of the modules, and add it again,
     with the same assignment and requirements.
  4. Verify that the Catalog dashboard loads without errors.

Change-Id: If021834d022661ef171167009b7eec79f3f37d83
Reviewed-on: https://gerrit.instructure.com/32870
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
add option to force re-login for oauth, fixes CAT-68

test plan:
- login to canvas
- do an oauth login from an external app, or with a manual GET request
  according to Step 1 of the "Web Application Flow" at
  https://canvas.instructure.com/doc/api/file.oauth.html,
  and make sure you include the query param force_login=1
- verify that canvas sends you to /login
- verify that without the param force_login=1, canvas still sends you
  directly to /login/oauth2/confirm

Change-Id: I49cd43cc48edf46f192221f5cf8707ee00e0bd8a
Reviewed-on: https://gerrit.instructure.com/32988
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeff Belser <jbelser@instructure.com>
Reviewed-by: Nick Houle <nhoule@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
adds completed_at to course_progress API

fixes CNVS-12257

test plan:
* set up a course with modules, module items
** and module completion requirements
* on each module, check the box that says:
** "students must move through requirements in
**  this module in sequential order"
* enroll a user in the course
* with that user's ACCESS-TOKEN, perform:

curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'include[]=course_progress' \
         <canvas>/api/v1/courses

* verify there is a completed_at field in the
** course_progress response
* verify that the completed_at field is nil unless
** the user has completed all the completion_requirements
** for all the modules
* if the user has completed all the completion_requirements
** for all the moduels, the completed_at date should be the
** date associated with the last thing the user did to complete
** the course.
* for example, if the last item is submitting an assignment,
** the completed_at date should be the submission date of that
** assignment
* verify that courses with 0 completion_requirements return
** nil for completed_at date

Change-Id: I6dfd6c7ec4168317cdcac0be13de52e9111f8c13
Reviewed-on: https://gerrit.instructure.com/32983
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Nick Houle <nhoule@instructure.com>
Reviewed-by: Dave Donahue <ddonahue@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
support for trusted developer keys

fixes CAT-66

Auto-authorize trusted developer keys during without prompting the end-
user. This will allow for more seamless integrations with other in-house
apps.

Also fix remember-me access so it works when you re-auth into canvas
(previously it only worked if you were already authenticated into
canvas)

Expand test coverage around oauth login scenarios

test plan setup:
* set up a web-based oauth integration with canvas
* ensure your app doesn't currently pass force_login=1 during oauth
* ensure your app doesn't currently delete access tokens during logout
* for part 2, ensure your app uses the /auth/userinfo scope (for part
  1 it doesn't matter)
* for part 3, ensure your app does token request flow (not userinfo)

test plan part 1 (trusted keys):
1. in the canvas console, set trusted=true on the app's developer key
2. do an oauth login as and end-user
3. confirm that you are authenticated into the app without being prompted
   to give it canvas access
4. log out of the app (but not canvas)
5. click to log in again
6. confirm that you are automagically logged in without any prompts

test plan part 2 (remember access):
1. in the canvas console, set trusted=false on the app's developer key (or
   set up your app to use a different one)
2. do an oauth login as and end-user
3. confirm that you are prompted to authorize the app
4. check the box to remember access
5. log out of canvas and the app
6. do an oauth login again
7. confirm you are not prompted to authorize the app
8. log out of the app (but not canvas)
9. click to log in again
10. confirm that you are automagically logged in without any prompts

test plan part 3 (untrusted key, not-userinfo)
1. in the canvas console, set trusted=false on the app's developer key (or
   set up your app to use a different one)
2. do an oauth login as and end-user
3. confirm that you are prompted to authorize the app
4. confirm there is no box to remember access
5. log out of canvas and the app
6. do an oauth login again
7. confirm you are prompted to authorize the app again
8. log out of the app (but not canvas)
9. click to log in again
10. confirm that are prompted to authorize the app again

Change-Id: Ifb2eb29e4da163b595cb070455ebae21a4618ba4
Reviewed-on: https://gerrit.instructure.com/32926
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
pass user info in custom_dashboard url

fixes CAT-70

test-plan:
* in rails console:
* a = Account.root
* a.settings[:dashboard_url] = "http://www.google.com"
* a.save
* login to canvas
* verify that you are redirected to "http://www.google.com?current_user_id=123"
* where current_user_id equals the id of the user you logged in as

Change-Id: I3074f821fa4454b2a568228d00b1da60324b71a7
Reviewed-on: https://gerrit.instructure.com/33225
Reviewed-by: Jon Jensen <jon@instructure.com>
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
warn teachers when changing grade display

fixes CNVS-11250

test plan:
  - as a teacher with unweighted assignment groups
  go to GB2 and toggle between points/percent
  - a modal should appear that warns you about this action
    - the text should make sense
    - it should say "a percent" or "points" depending on
      your current settings
    - clicking cancel or the x should close the dialog
    - clicking continue should close the dialog and change the totals
      - when you reload the page, totals should remain changed
    - clicking the checkbox and then continue should do the same
      - however the next time you click the dropdown, the modal should
        not appear, even after reload
        (other courses/browsers will still show this user the modal)
    - as a TA or another teacher in the same class, the modal should
      still appear (until you click the checkbox)

Change-Id: I465bd7b9d770077931a769efb7904c1349920a3c
Reviewed-on: https://gerrit.instructure.com/33356
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Amber Taniuchi <amber@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Product-Review: Hilary Scharton <hilary@instructure.com>
spec: fix fragile redis delayed_job specs

Jobs on separate strands inserted at very close to the same time might
get processed in a different order, due to the way that redis uses
timestamps for ordering rather than database ids.

This can also change which of the two calls in some specs return the
job, and which don't, after manual redis manipulation.

Change-Id: I2029ccd4ddd273a98b9bb7fdc19ccfc5cfd9e488
Reviewed-on: https://gerrit.instructure.com/33733
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
QA-Review: Brian Palmer <brianp@instructure.com>
Updated pl translation
pass non-urlified wiki page names when redirecting

test plan:
* use the wiki sidebar to create a "Link to a New Page"
* use unicode characters in the title
* click on the link
* create a new page
* confirm that the title of the new page is the same as
 the original title you entered

fixes #CNVS-12136

Change-Id: Id2dc1bb72aefa814d840f4b7fd75bcef260eda61
Reviewed-on: https://gerrit.instructure.com/33606
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
Product-Review: Jeremy Stanley <jeremy@instructure.com>
QA-Review: Clare Strong <clare@instructure.com>
bump rake gem

Change-Id: I751f667195264b4066ed852767c2240b4ef0db6f
Reviewed-on: https://gerrit.instructure.com/33642
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Brian Palmer <brianp@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
QA-Review: Cody Cutrer <cody@instructure.com>
move find_or_create_submission to own object

closes CNVS-12576

test plan:
  - create a quiz as an instructor
  - as a student
    - take quiz
      - exercise all paths by which a quiz submission is generated
    - verify that quiz submissions continue to work as before

Change-Id: I4c97162589b5fb9cceb0c6cbe9e4fb68dd1f8093
Reviewed-on: https://gerrit.instructure.com/33691
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Josh Simpson <jsimpson@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Product-Review: Jason Madsen <jmadsen@instructure.com>
add migration external tool extension ui

An external tool configured for the migration extension will
show up on a course's content import page and allow you to
select a file to migrate from the LTI launch.

fixes LOR-20

Test Plan:
 * Configure an external tool that has the migration_selection
   extension
 * The new tool should show in the dropdown on the content
   import page.
 * Run some migrations through the external tool.

Change-Id: Iea16a7e32145e6799c551816e6f70a73f2a8c3d0
Reviewed-on: https://gerrit.instructure.com/21443
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jake Trent <jaketrent@instructure.com>
Reviewed-by: Brad Humphrey <brad@instructure.com>
QA-Review: Amber Taniuchi <amber@instructure.com>
Product-Review: Jon Willesen <jonw@instructure.com>
add migration_selection lti extension

this commit lets canvas remember that an LTI has this extension.
launching and using LTI for content migrations will happen
separately.

refs LOR-20

test plan:
 - configure an LTI tool with XML that includes the
   migration_selection extension
 - the tool should show up in the tool list with the "Migration
   Selection" tag.

Change-Id: I880125515a32af1b9a39fbd1787617bceae74f50
Reviewed-on: https://gerrit.instructure.com/21442
Reviewed-by: Jake Trent <jaketrent@instructure.com>
Reviewed-by: Bracken Mosbacker <bracken@instructure.com>
Reviewed-by: Brad Humphrey <brad@instructure.com>
QA-Review: Amber Taniuchi <amber@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Jon Willesen <jonw@instructure.com>
spec: fix redis job specs in rails3

This class doesn't rely on ActiveRecord::Base in rails3, and the ==
method needed to be a bit smarter.

Change-Id: I64fc2b58fc29ebb85772567634c5ea3c4c41b9fb
Reviewed-on: https://gerrit.instructure.com/33754
Reviewed-by: James Williams  <jamesw@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: James Williams  <jamesw@instructure.com>
QA-Review: James Williams  <jamesw@instructure.com>
Remove unused code from Linkedin module

fixes: CNVS-12494

test plan:
* regression test for linkedin

Change-Id: Ibf6c78d767301184bc487813384f7c33ad78997a
Reviewed-on: https://gerrit.instructure.com/33428
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Simon Williams <simon@instructure.com>
QA-Review: Trevor deHaan <tdehaan@instructure.com>
Product-Review: Jason Madsen <jmadsen@instructure.com>