# Presentation: 934813

## DATA INTEGRITYWORKGROUPActivities and Recommendations

- JOB CORPS LEADERSHIP CONFERENCE

## DIG Membership

- Peter Calvo, Chair
- Mary Cohorst
- Wendy Boyd
- Dennie Moore
- Sue Larson
- Jim Cooper
- Pat Putnins
- Chris Conboy
- Andrew Bauck

- Bob James
- Dennis Johnson
- Lynn Kotecki
- Dale Fischer
- Melanie Walker
- Dave Henn
- Scott Ruane
- Jimmy McDonough
- Pam Knight

## PY-05 DIG Charge

- Amend the PAG to include mandatory activities for testing the validity and reliability of performance outcomes
- Retain records that document the testing performed
- Take appropriate action to recover any overpayments made to the contractor as a result of misreported performance data

## PY-05 Activities

- JCIN 04-14 changed the PAG to require Regions to conduct mandatory audits of 10% of all student records associated with performance outcomes during all center assessments
- Battelle/DIR worked with the DIG to develop targeted samples to assist Regional Offices in the conduct of data integrity audits

## PY-05 Activities, continued

- In consultation with the FMT and NJCA, the committee rolled-out PRH Change Notice 05-13, which outlined requirements for:
- Contractor and Agency Quality Assurance Plan submission
- Regional requirements for the conduct of program assessments
- Use of targeted sample audits during assessments
- Documentation of results of program assessments and audits
- Assessment of liquidated damages for instances of misreporting
- Recovery of liquidated damage determinations and removal of credits in the Job Corps performance management system
- Notification to the National Office and OIG for instances of apparent fraud

## Steps For Assessing Liquidated Damages

## Steps For Assessing Liquidated Damages

- RO conducts Data Integrity assessment
- CO issues Initial Notice of Liquidated Damages
- Contractor submits appeal
- CO reviews appeal
- CO issues Final Determination of Liquidated Damages
- Data Center removes invalid credits
- Contractor identifies liquidated damages through fee credit on 2110
- RO deducts (reconciles) liquidated damage assessment through PBSC fee reconciliation

**Manual Procedure Overview**

## Steps For Assessing Liquidated Damages

- The Federal Acquisition Regulation (FAR) requires that a contractor provide and maintain an inspection system acceptable to the Government covering the services under the contract.  Records of all work performed by the contractor must be maintained and made available to the Government during contract performance and for as long afterwards as the contract requires.
- The Government has the right to inspect and test all services called for by the contract.
- If any of the services performed do not conform to contract requirements, the Government may require the contractor to perform the services again in conformity with contract requirements, for no additional fee.  When the defects in services cannot be corrected by re-performance, the Government may (1) require the contractor to take necessary action to ensure that future performance conforms to contract requirements and (2) reduce any fee payable under the contract to reflect the reduced value of the services performed.

**JC Procurement Compendium**

## Initial Notice of Liquidated Damages

- RO issues within 45 days of assessment
- Notice Content:
- Identifies Data Integrity findings by type
- Establishes liquidated damage charges
- Requests corrective actions
- Lists individual student records & findings
- Describes appeal procedures
- Sets appeal timeline (60 days)
- Identifies appeal format

## OPERATOR APPEALS

**Timeline**** **- Must be received within 60 days of Initial Notice
**Content** -
- Describes corrective actions taken
- Identifies areas of agreement & disagreement
- Asserts justification & rationale for each finding being appealed
- Provides documentation supporting assertions (GEDs, HSDs, TARs etc.)
**Format** – Submitted using Appeal/Decision Template

## CO FINAL DETERMINATION

**Timeline**** **- Must be issued within 60 days of appeal receipt
**Content (per FAR)** -
- Identifies initial findings
- Summarizes the operator’s appeal
- Identifies areas of agreement & disagreement
- States CO’s final decision on each finding
- Summarizes Data Integrity findings by type
- Sets final liquidated damage amount due
- Issues a demand for payment & procedures
- Instructs contractor to post a credit to the fee line of the contract expense report
- Lists credits to be removed from data system
- Identifies FAR procedures for subsequent contractor claims
- Board of Contract Appeals
**Format** – Documented using Appeal/Decision Template

## Liquidated DamagesAppeal/Decision Template

**Contractor Accepts **
**Or Appeals Initial **
**Findings**

**Contractor Enters**
**Reason for Appeal**

## Liquidated DamagesAppeal/Decision Template

**Region Enters**
**Decision on Appeal**

**Credits to be **
**Removed**

## Modify 2110 & 2181Budget & Expense Reports

**Fee currently shown as a lump sum**

**Credit for liquidated damages. Purpose is unidentified**

## Incentive Fee Reconciliation Worksheet

**Liquidated Damage Amount**

**Base & Incentive Fee Amounts**

## PY-06 DIG Charge

- Complete and publish revised PAG
- Develop and implement staff training to promote full implementation of the revised PAG
- Develop additional tools in collaboration with JCDC to assist in reporting and tracking data anomalies
- Develop, test, and implement additional audit strategies based on data anomalies
- Analyze results of audits and assessments, and make recommendations to improve program outcomes (data-based decision making and policy revisions)

## DIG Training

- The committee, in collaboration with Battelle and DIR, developed and tested a Data Integrity Audit Guide
- Training and on-site Technical Assistance on DIG audits was provided by DIR and Battelle
- PAG training was provided during the Project Manager’s Training in Chicago
- JCDC released updates for the DIG tools in the JC Suite of Applications (EIS & CIS)

## Additional Testing & Tools

- FMS
- The committee is working with the Data Center on the development of an automated PBSC fee reconciliation
- EIS
- Additional Tools are being developed to evaluate LOS for trades across Regions to better gauge rigor and identify anomalies
- CIS
- Enhancements are being proposed to provide more management reports that promote outcome improvement through trend analysis (CTT Completion & OA Outcomes)

## Additional Tools

- Executive Information System and CIS
- In collaboration with JCDC, the committee provided requirements for the implementation of various tools in EIS and CIS
- Short/Timing of Vocational Completions
- Short/Timing of GEDs
- Excess AWOL and Extended Enrollment
- Targeted Samples
- In collaboration with DIR/Battelle, the committee provided requirements and instructions for implementation of Targeted Audit Samples for Center DIG Assessments

## Improving Program Outcomes

- The committee is considering the a new suite of applications that would assist Regions in developing the assessment schedule, program monitoring and assessment of liquidated damages, PBSC fee reconciliations, and serve as a clearinghouse for those various documents
- The system, similar to JCRL, will hold templates for Assessment notifications, pre-onsite performance analysis, Targeted samples, and other items

## FMT Recommendations

**Adopt Revised PAG**
- Tool for Compliance, Quality, and Data Integrity Assessment of JC programs
**Adopt Policy Changes for Placement**
- Clarifies Placement Definition, Eliminates Potential Misreporting, and Promotes Services to Students
**Adopt Policy Changes for Leave and Attendance**
- Reduces Loss Time, Promotes Career Success Standards and Orderly Separations
**Adopt CTT/WBL Policy Changes**
- Promotes Industry Certification within and across Clusters and Eliminates Potential Misreporting

## Recommendation #1

**Adopt the Revised PAG**
**Standardize assessment procedures and strategies across Regions**
**Place emphasis on both quality and compliance**
**Includes tools to assess data integrity**
**Expands adjectival rating to mirror CPER**
**Piloted in numerous assessments**
**Training already provided to Project Managers**
**The change is needed to address Data Integrity concerns expressed by the OIG.  **

## Recommendation #2

**Adopt Policy Changes for Placement**
- Definition of Placement Dates
- Elimination of Date Reported to reduce potential for misreporting.  CTS must report placement within 15 days of meeting placement definition.
- Placement Definitions
- Redefine the definitions to eliminate confusion and align with 3rd party verification providers.
- 40 hours or more in one or more jobs in a 14 consecutive day period.
- Verification
- Clarifies Policy language with respect to verification documentation (Forms) in Exhibit 4-2 to reduce potential misreporting.
- Re-verification
- Modified language to match “questionable placement” activities verifying original placement verification documentation.
- Wage
- Tips and commissions are obtained directly from the employer and used to determine the average wage and added policy to indicate OT is not part of the average wage.

## Recommendation #3

**Adopt Policy Changes for Leave and Attendance**
**Clarify documentation and verification requirements to reduce potential misreporting or abuse.**
**Reduce the amount of time students are not in training and improve orderly separations (60+ to 37 with 12 earned).**
**Align leave policy and language with the workplace to promote CSS **
**Revise PRH 6, and Exhibits 6-1, 6-2, and 6-3**
**Define use, limitation, verification, and documentation requirements**
**Modify Leave Categories to align with workplace by reducing entitlement leave and promoting earned leave.  **
**Add categories to define the various WBL/CBL activities**
**Reduce recordkeeping burden and improve accountability**

## Summary of Leave Changes

- P-DON
- Added P-DON for non-res students in ACT/OCT
- P-DOF
- Expanded categories to account for various movements
- P-DOF, Home-Based Work Based Learning (6 weeks)
- P-DOF, Off-Center CTST (formerly VST)
- P-DOF, Job Search (10 days throughout enrollment)
- P-DOF, Fire/National Emergency (8 weeks)
- P-DOF, Other (10 Days – Recruiting)

## Summary of Leave Changes

- P-EMLV, Emergency Leave eliminated
- P-ADLV, Paid Administration Leave
- Reduced from 10 to 5 days every 6-months
- Clarified use of category and documentation requirement
- Removed “Other circumstances of an urgent personal nature”
- P-BERV, Bereavement Leave
- Replaces Emergency Leave and Clarifies its use
- 10 days per year with verification

## Summary of Leave Changes

**P-PLV, Paid Personal Leave (New)**
- To be used at a student’s discretion based upon center approval
- 5 training days available upon enrollment
- After 6 months of enrollment, 1 personal day is accrued every 30 paid days of enrollment – if a student has any JA days during the 30 days they do not accrue a personal day
**N-JA, Job Abandonment (AWOL)**
- 5 consecutive training days or 10 days in a 6-month period.  Students will be terminated on the 6th and 11th day
- Students will forfeit $250 of their transitional payment

## Summary of Leave Changes

- N-UPAL, Unpaid Admin Leave
- Clarified uses and documentation requirements
- Reduced to 5 training days per 6 month period, based on enrollment (from 30 per year)
- Regions can authorize additional 5 UPAL days for students who have extenuating circumstances such as subsistence fishing/hunting, or religious accommodations.
- Documentation of the need for student’s presence at home and RO approval to be maintained in the permanent record.

## Summary of Leave Changes

- N-UPER, Unpaid Personal Leave (New)
- For custodial parents who have to remain at home to care for children during sickness or interruption in day care services.
- Unpaid personal leave day must be requested by the student no later than 1 hour past scheduled arrival time, on the date of request
- No special documentation required
- 5 training days available upon enrollment
- Can only be used after paid personal days are exhausted

## Other Considerations

- Elimination of Temporarily Unaccounted For (TUF)
- $250 deduction in transitional payment for AWOL – Job Abandonment Separation
- $500 deduction in transitional payment for discipline separation
- The FMT is also looking at expanding transitional payments up to $2,000 in support of JTM placements, Retention, and OMS measures

## Recommendation #4

**Adopt CTT/WBL Policy Changes**
- Clarify TAR recordkeeping requirements to reduce potential misreporting or abuse.
- Promote WBL as a means to capitalize on the President’s High Growth Job Initiative
- Modify and clarify P-DOF definition
- Standardize TAR recordkeeping language
- Enable assignment of concurrent industry cluster Career Technical Training assignments

## Proposed Language (CTT)

- PRH 3.13 R3.  Program Administration
- 	a.  Students may be enrolled in one or more career technical education programs at any given time.  The center must ensure students are assigned to the applicable O*Net(s) for those programs assigned in CIS.
- 	b.  Enrollment in various levels of a given industry cluster need not be sequential. TARs may be taught concurrently, regardless of the start date of each of each individual TAR.  Completion level TARs may be taught concurrently with Foundation Skills; however, credit cannot be granted for the completion level until the Foundations skills are certified.

## Proposed Language (CTT)

**PRH 3.13 R6. Reporting/Documentation/Record Keeping **
- Centers shall ensure that all student progress is documented on TARs.
- Student progress must be documented at least weekly, when the student is assigned and present for CTE instruction.
- The date of the weekly progress update may be used as the date in which credit is given for all skills acquired since the last update. (e.g. skills learned during the past week).
- Each TAR competency, or group of competencies, shall require the initials of the instructor and the student as a means to verify skill mastery and completion status.  The date of the progress update must be included on the TAR by the initials.  Corrections require the crossing out and initialing of the change(s).
- Completed TARs will be credited in CIS only after the TAR has been certified by Center Management.  A TAR is considered certified when it is audited by a designated center manager and signed indicating competencies of the trade had been met.
- Original TAR(s) must be included in the student’s permanent record prior to crediting the student with completion or at separation for non-completers.

## Next Steps - Policy

- The proposed policy changes are being reviewed by the Federal Management Team (FMT)
- After review by the FMT, the proposed policy language will be sent out for public comment
- The proposed policy language changes are anticipated to be approved prior to the winter break

## Next Steps - Committee

- The Committee plans to:
- Promote CIS/EIS changes recommended for CTT
- Promote enhancements to FMS in order to modify CPER and add automated PBSC reconciliation
- Promote continued enhancements proposed for CIS and EIS to develop tools and resources necessary for data integrity and continuous program improvement

- QUESTIONS?