don't use scope.includes(:user) cross-shard

fixes CNVS-12038

scope.includes(...) doesn't know about sharding. preload_associations
does know about sharding, and will make sure to load all records on any
existing associated shards, handle missing shards, etc., and in at most
one more query than includes(...) would have used

test-plan:
 - load an account's settings page. it still works
 - add a cross-shard account user to that account. still works
 - change the cross-shard AccountUser's user_id to an id from a
   non-existent shard. create a shadow user record on the account's
   shard with that new user id. e.g. in script/console:

     account_user = AccountUser.find(...)
     account_user.user_id = 1_230_000_000_000_001
     account_user.save
     User.connection.execute("INSERT INTO users (id) VALUES (1230000000000001)")

   account settings page should still work (this is what broke
   CNVS-12038)

Change-Id: I38907aa1059ed5ed98f70559ecc16a430372988d
Reviewed-on: https://gerrit.instructure.com/32674
Reviewed-by: Rob Orton <rob@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Cody Cutrer <cody@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Jacob Fugal <jacob@instructure.com>
fix bad date math in gradebook dialog

fixes CNVS-12386

test plan:
- setup some assignment with submissions in different combinations of:
  - late / on-time
  - single submission / multiple submissions
- make sure that GB2 correctly differentiates between late and on time in the
  submission details dialog (that pops up when you click in the top right
  corner of a cell)
  - it will be text for a single submission and dropdown for multiple

Change-Id: Ib5f62dd95b073461cfcbea85b1ed8e0805f2a57f
Reviewed-on: https://gerrit.instructure.com/33263
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Mike Nomitch <mnomitch@instructure.com>
QA-Review: Amber Taniuchi <amber@instructure.com>
Product-Review: Bryan Madsen <bryan@instructure.com>
remove OpenDataExport permissions/routes

the gem will handle the process of including them

Change-Id: I2237b4cad388756f91f67138a716ae5a8472d9df
Reviewed-on: https://gerrit.instructure.com/32586
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Nick Cloward <ncloward@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Anthus Williams <awilliams@instructure.com>

Conflicts:
	config/routes.rb
use the current time for grade change audit events

fixes CNVS-12400

test plan: Enable auditors and trigger some grade change events, such as
auto-graded quiz or teacher-graded submission. They should show up in
the audit logs with the timestamp of the time the event happened.

Change-Id: I596242e09d8acb34443ca0974b7393fc9faa51f6
Reviewed-on: https://gerrit.instructure.com/33342
Reviewed-by: Cody Cutrer <cody@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: August Thornton <august@instructure.com>
QA-Review: Jeremy Putnam <jeremyp@instructure.com>
Product-Review: Brian Palmer <brianp@instructure.com>
don't show rubric assess scores in api on muted assignments

fixes CNVS-12403

test plan:
- create an assignment with a rubric, and mute it
- grade the assignment with the rubric for a student
- as the student, query the api for the submission object
- it should not contain the rubric assessment data

Change-Id: Id83a9444f06c59938cd1bd5cac7a6ddb4019d45f
Reviewed-on: https://gerrit.instructure.com/33316
Reviewed-by: Liz Abinante <labinante@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Amber Taniuchi <amber@instructure.com>
Reviewed-by: Cameron Matheson <cameron@instructure.com>
Product-Review: Cameron Matheson <cameron@instructure.com>
don't localized datetimes formatted for datepicker

refs CNVS-12398

the eventual complete solution will involve teaching the date picker to
parse the localized datetime, but for now, just format the time in
english if the target is a datepicker's field value.

test-plan:
 - set locale to portuguese
 - datepickers (e.g. assignment due date when editing) populated on page
   load should contain the date time in english (e.g. "<date> at 3:00pm"
   vs. "<date> em 3:00")
 - using the date picker should produce a time in english in the field
 - saving the english time should work
 - non-datepicker field values with the same formatting rules (e.g.
   assignment due date when *not* editing) should display localized

Change-Id: Ibc69d2066518b4875c1f42b794df784c7e70c2c7
Reviewed-on: https://gerrit.instructure.com/33444
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Reviewed-by: Anthus Williams <awilliams@instructure.com>
Product-Review: Rob Orton <rob@instructure.com>
add serializable hash of CustomData for User models

fixes CNVS-11424

test plan:
- rake db:migrate
- if you don't already have one, set yourself up with a developer token.
  (you can do so from <canvas>/developer_keys)

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data
  #=> {"message":"no data for scope"}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X PUT -F 'ns=test' \
         -F 'data[apple]=so tasty' \
         -F 'data[kiwi]=a bit sour' \
         <canvas>/api/v1/users/self/custom_data/fruit
  #=> {"data":{"apple":"so tasty","kiwi":"a bit sour"}}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data
  #=> {"data":{"fruit":{"apple":"so tasty","kiwi":"a bit sour"}}}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X DELETE -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data/fruit/kiwi
  #=> {"data":"a bit sour"}

- $ curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'ns=test' \
         <canvas>/api/v1/users/self/custom_data
  #=> {"data":{"fruit":{"apple":"so tasty"}}}

- see new API doc for more info about how it should work, but
  basically, you should be able to GET, PUT, and DELETE at will
  for any given scope, and PUTting JSON hashes creates referenceable
  scopes. (e.g. the DELETE above has 'kiwi' in its scope)

Change-Id: If027ae4aeec14edf44275ba0372a68aef7e5600e
Reviewed-on: https://gerrit.instructure.com/31173
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Matt Fairbourn <mfairbourn@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Jon Jensen <jon@instructure.com>
course progress api

fixes CNVS-11526

test plan:
* set up a course with modules, module items
** and module completion requirements
* on each module, check the box that says:
** "students must move through requirements in
**  this module in sequential order"
* enroll a user in the course
* with that user's ACCESS-TOKEN, perform:

curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'include[]=course_progress' \
         <canvas>/api/v1/courses

* verify that in the response for all the courses
** for that user, there is a "course_progress"
** object.
* verify that the "requirement_count" is the
** total number of requirements for all the modules
** combined.
* verify that the "requirement_completed_count" is
** the number of requirements completed by this user.
* verify that the "next_requirement_url" is the url
** to the module item for the next requirement the
** user needs to complete for the course
* verify that the "next_requirement_url" is null
** if the user has completed all the course's
** requirements
* verify that "course_progress" returns an error message
** if the course is not module based (has no modules)

example:
"course_progress": {
  "requirement_count": 23,
  "requirement_completed_count": 1,
  "next_requirement_url":
    "http://localhost/courses/23/modules/items/31"
}

in this example, there are 23 total requirements for
the course. the current user has completed 1. the
"next_requirement_url" is a link to the next module
item in the sequence that still does not have its
requirement(s) completed by the current user.

Change-Id: I05897a263ea68a72c1583a9e5c19e32b685ec037
Reviewed-on: https://gerrit.instructure.com/31616
Reviewed-by: Jon Jensen <jon@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Matt Fairbourn <mfairbourn@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
bugfix for course progress api (fixes CNVS-12106)

Issue:
  * When creating a module, then a new assignment under that module,
    the ContextModuleProgression.requirements_met field comes back
    as nil, causing course_progress_json to bomb. The previous code
    assumed an array would always be returned, but this is not the case.

Solution:
  * Cast requirements_met to an array.
    + nil.to_a -> []
    + [1].to_a -> [1]

Test plan:
  * Test against Gallery course progress commit (g/32571) with some new modules
    and blank assignments created through the module interface.

Change-Id: Ie666fcc0031009b5ff32ad151659cb2818bf7b6e
Reviewed-on: https://gerrit.instructure.com/32570
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
Reviewed-by: Michael Ziwisky <mziwisky@instructure.com>
course progress API fixes (CNVS-12156, CNVS-12157)

Fixes CNVS-12156

Problem:
  Removing a module item (ContentTag) should update the completion
  requirements on its ContextModule. This is not directly related
  to the course progress API, but rather a separate bug deeper
  in the code that is affecting the API response.

Test plan:
  1. Add an assignment to a module, and set completion requirements
     on the module.
  2. Assert that the completion requirements are set:
       ContextModule.find_by_name('Module name').completion_requirements
  3. Remove the assignment from the module through the UI.
  4. Assert that the completion requirements no longer reference the
     old assignment:
       ContextModule.find_by_name('Module name').completion_requirements

Fixes CNVS-12157

Problem:
  Obsolete completion requirements were tripping up the progress API
  response. For instance, turning in an assignment that used to be a
  requirement but no longer is would have an impact on the
  requirement_completed_count, sometimes causing it to be larger
  than requirement_count, leading to completion percentages greater
  than 100.

Test plan:
  1. As a student, complete a required assignment on a module.
  2. Using your favorite LTI tool that uses the progress API (e.g. Catalog),
     check the course completion percentage.
  3. As an admin, remove the completion requirement for that assignment.
  4. Check the course completion percentage again and verify that it has
     decreased.

Change-Id: I80a932657eada20717ca2e0df7b57a005a5b276a
Reviewed-on: https://gerrit.instructure.com/32698
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
Reviewed-by: Jeremy Stanley <jeremy@instructure.com>
add option to logout on oauth token deletion, fixes CAT-67

test plan:
- do an oauth login, perhaps by using some external app, or by following
  the guide at https://canvas.instructure.com/doc/api/file.oauth.html
- end the oauth session with param expire_sessions=1, i.e.
    DELETE <canvas>/login/oauth2/token?expire_sessions=1
- do another oauth login, and notice that you're asked for your
  credentials.  i.e., your canvas session was destroyed.  that's a good
  thing.

Change-Id: I765d8f62dfcd133bc652b484c26c7bb1fc86e24a
Reviewed-on: https://gerrit.instructure.com/32866
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Course progress API returns 500 on changed requirements

Fixes CNVS-12210

The course progress API currently works as follows:

- Fetch all the modules for this user
- Get a list of the completion requirements
- Cross-check the requirements against the last known
  list of requirements met
- If there are outstanding requirements, refresh progression
  data, and fetch the URL of the next item

The problem is that in that last step, we refresh the progression
data, which means that the dataset changes mid-method, and it can
cause unexpected behavior.

The solution is to refresh progression data as step 2, so that
the entire method is looking at the same information.

Test plan:
  1. As a teacher, add a couple of modules with completion requirements.
  2. As a student, complete the requirements.
  3. As the teacher, remove one of the modules, and add it again,
     with the same assignment and requirements.
  4. Verify that the Catalog dashboard loads without errors.

Change-Id: If021834d022661ef171167009b7eec79f3f37d83
Reviewed-on: https://gerrit.instructure.com/32870
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
add option to force re-login for oauth, fixes CAT-68

test plan:
- login to canvas
- do an oauth login from an external app, or with a manual GET request
  according to Step 1 of the "Web Application Flow" at
  https://canvas.instructure.com/doc/api/file.oauth.html,
  and make sure you include the query param force_login=1
- verify that canvas sends you to /login
- verify that without the param force_login=1, canvas still sends you
  directly to /login/oauth2/confirm

Change-Id: I49cd43cc48edf46f192221f5cf8707ee00e0bd8a
Reviewed-on: https://gerrit.instructure.com/32988
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jeff Belser <jbelser@instructure.com>
Reviewed-by: Nick Houle <nhoule@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
adds completed_at to course_progress API

fixes CNVS-12257

test plan:
* set up a course with modules, module items
** and module completion requirements
* on each module, check the box that says:
** "students must move through requirements in
**  this module in sequential order"
* enroll a user in the course
* with that user's ACCESS-TOKEN, perform:

curl -H "Authorization: Bearer <ACCESS-TOKEN>" \
         -X GET -F 'include[]=course_progress' \
         <canvas>/api/v1/courses

* verify there is a completed_at field in the
** course_progress response
* verify that the completed_at field is nil unless
** the user has completed all the completion_requirements
** for all the modules
* if the user has completed all the completion_requirements
** for all the moduels, the completed_at date should be the
** date associated with the last thing the user did to complete
** the course.
* for example, if the last item is submitting an assignment,
** the completed_at date should be the submission date of that
** assignment
* verify that courses with 0 completion_requirements return
** nil for completed_at date

Change-Id: I6dfd6c7ec4168317cdcac0be13de52e9111f8c13
Reviewed-on: https://gerrit.instructure.com/32983
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Nick Houle <nhoule@instructure.com>
Reviewed-by: Dave Donahue <ddonahue@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
support for trusted developer keys

fixes CAT-66

Auto-authorize trusted developer keys during without prompting the end-
user. This will allow for more seamless integrations with other in-house
apps.

Also fix remember-me access so it works when you re-auth into canvas
(previously it only worked if you were already authenticated into
canvas)

Expand test coverage around oauth login scenarios

test plan setup:
* set up a web-based oauth integration with canvas
* ensure your app doesn't currently pass force_login=1 during oauth
* ensure your app doesn't currently delete access tokens during logout
* for part 2, ensure your app uses the /auth/userinfo scope (for part
  1 it doesn't matter)
* for part 3, ensure your app does token request flow (not userinfo)

test plan part 1 (trusted keys):
1. in the canvas console, set trusted=true on the app's developer key
2. do an oauth login as and end-user
3. confirm that you are authenticated into the app without being prompted
   to give it canvas access
4. log out of the app (but not canvas)
5. click to log in again
6. confirm that you are automagically logged in without any prompts

test plan part 2 (remember access):
1. in the canvas console, set trusted=false on the app's developer key (or
   set up your app to use a different one)
2. do an oauth login as and end-user
3. confirm that you are prompted to authorize the app
4. check the box to remember access
5. log out of canvas and the app
6. do an oauth login again
7. confirm you are not prompted to authorize the app
8. log out of the app (but not canvas)
9. click to log in again
10. confirm that you are automagically logged in without any prompts

test plan part 3 (untrusted key, not-userinfo)
1. in the canvas console, set trusted=false on the app's developer key (or
   set up your app to use a different one)
2. do an oauth login as and end-user
3. confirm that you are prompted to authorize the app
4. confirm there is no box to remember access
5. log out of canvas and the app
6. do an oauth login again
7. confirm you are prompted to authorize the app again
8. log out of the app (but not canvas)
9. click to log in again
10. confirm that are prompted to authorize the app again

Change-Id: Ifb2eb29e4da163b595cb070455ebae21a4618ba4
Reviewed-on: https://gerrit.instructure.com/32926
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Jon Jensen <jon@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
pass user info in custom_dashboard url

fixes CAT-70

test-plan:
* in rails console:
* a = Account.root
* a.settings[:dashboard_url] = "http://www.google.com"
* a.save
* login to canvas
* verify that you are redirected to "http://www.google.com?current_user_id=123"
* where current_user_id equals the id of the user you logged in as

Change-Id: I3074f821fa4454b2a568228d00b1da60324b71a7
Reviewed-on: https://gerrit.instructure.com/33225
Reviewed-by: Jon Jensen <jon@instructure.com>
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Tested-by: Jenkins <jenkins@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>
hide all quiz questions from students in the questions api endpoint

fixes CNVS-12583

test plan:
- restricting access until after a quiz begins
  - as a teacher
    - create a quiz with a question
  - as a student
    - in all cases you should *not* be able to see the quiz questions in the api

Change-Id: Ide5cf4f31ed27fd0f1addf5c47a0d6f5cab0efa2
Reviewed-on: https://gerrit.instructure.com/33743
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Josh Simpson <jsimpson@instructure.com>
Product-Review: Derek DeVries <ddevries@instructure.com>
fix js error you sometimes get when editing an appointment group

fixes CNVS-12558

test plan:
 * enable calendar2 and scheduler
 * go to calendar, then scheduler view
 * ensure you have an appointment group (if not, create one)
 * ensure appointment_group_id=null is *not* in the url
 * click the appointment group to view the calendar for it
 * appointment_group_id=x should be in the url
 * click "Go back to list"
 * appointment_group_id= should *not* be in the url
 * reload the page
 * try to edit the appointment group
 * it should work

Change-Id: Idc15a14a1957b8322b1f48ee577770072309b114
Reviewed-on: https://gerrit.instructure.com/33647
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Nick Cloward <ncloward@instructure.com>
QA-Review: August Thornton <august@instructure.com>
Product-Review: Cody Cutrer <cody@instructure.com>
fix failure of quiz submissions to merge properly in user merge

fixes CNVS-12512

test plan:
  - create two users and enroll them in the same course
  - create a quiz in the course and have User_B take the quiz
  - using the user merge api:
    https://canvas.instructure.com/doc/api/users.html#method.users.merge_into
  - merge User_B into User_A, so that User_A will be the new primary user
  - check the quiz submissions and notice that the user should have submissions

Change-Id: I8c91d61b418ed486956d13f334f71684fa95cfb5
Reviewed-on: https://gerrit.instructure.com/33788
Tested-by: Jenkins <jenkins@instructure.com>
QA-Review: Caleb Guanzon <cguanzon@instructure.com>
Reviewed-by: Josh Simpson <jsimpson@instructure.com>
Product-Review: Derek DeVries <ddevries@instructure.com>
customize "Canvas" in registration confirmation email, fixes CAT-128

add a product_name account setting to allow schools to customize the
registration confirmation email. ostensibly this setting could be used
throughout canvas to allow greater customization, but let's not get ahead
of ourselves

test plan:
1. on your root account, set settings[:product_name] to something
2. /register as a new user
3. confirm that the "set a password" email uses that product name in the
   header and body

Change-Id: I6e0d5aa8ad80c4cda40e2aac1367364293843627
Reviewed-on: https://gerrit.instructure.com/33846
Tested-by: Jenkins <jenkins@instructure.com>
Reviewed-by: Dave Donahue <ddonahue@instructure.com>
Product-Review: Marc LeGendre <marc@instructure.com>
QA-Review: Marc LeGendre <marc@instructure.com>