# Presentation: 243847

## Chapter 6.Making Success Measurable Through Quality Control

## Chapter 6.  Section 1.  Introduction to Quality Control

## Introduction

- How do you know the quality and accuracy of your rent calculations?
- Wait for HUD to tell you?
- Close your eyes and hope for the best?
- Assume that good people lead to good outcomes?
- Wait until you tally SEMAP scores?
- Or .....

## Introduction

- Are you interested in a better way that could:
- Enable you know when to take action on performance issues,
- Help document the improvements in error reduction you are making,
- Enhance the quality of the services you provide?

## Introduction

- A good quality control program can be that better way that helps you:
- Prevent errors
- Documents compliance with program requirements
- Demonstrate performance improvement over time

## Introduction

- An effective quality control program:
- Facilitates conversation about expectations and performance for individuals and the organization as a whole.
- Enables program managers to be proactive, rather than reactive

**Notes:**
Trainer note: Give a non-housing example, like this: 
Error: Late for training
Understand the Requirements: Be on time
Actual Performance: 15 minutes late
Reasons: Stopped for coffee and line was longer than I thought
Possible Solutions.... Bring thermos, skip the coffee (bad idea...) come earlier... take subway... have a friend buy my coffee... Select a solution: Take Metro
Coordinate: Metro doesn’t work with day-care
Go back and select again: Bring a thermos (Added advantages: better quality coffee, and cheaper)
Implementation plan: Tomorrow morning, bring coffee. Need 15 minutes to make coffee. Coordinate with other activities: I can make coffee while I feed the dog – I only need 5 extra minutes. Set the alarm and get out of bed!

Trainer Note: Emphasize that the process can be iterative/circle back to earlier points

Trainer Note: Quick brainstorm: “What will be the most challenging part of an error reduction initiative at your PHAs?”

## Introduction

- In this session, we will talk about:
- Three types of quality control
- Illustrate the thought process needed to  develop a QC program
- Provide sample QC tools
- Provide guidance on communicating with  your staff about quality issues.

## Introduction

- We will focus on HUD’s expectations related to income and rent determination
- BUT, the QC concepts we discuss apply equally well to other parts of a PHA operation.

## Why Quality Control?

- Quality control can help to confirm what you already know.  It can:
- Provide documentation that supports  recommendations you want to make,
- Generate statistics that document staff performance,
- Provide valuable information on a systematic basis to share within and outside your organization.
- .

## Why Quality Control?

- QC can be even more helpful if the results are a surprise. Then the results may highlight:
- Problems of which you were not aware
- Good news about improved performance or excess capacity.
- Were there aspects of the RIM review for your agency that were a surprise?

## Chapter 6.  Section 2.Types of Quality Control

## Types of Quality Control

- Continuous

- Periodic

**Notes:**
We will be talking about three different types of Quality.  It’s important to understand the differences and what each type offers. BUT, the point is not to learn to classify QC activities but for you to choose the QC techniques that best fit your needs.  Continuous QC is at the top of the triange because we think this is the approach that can have the most impact on error reduction.  But we will talk briefly about the other two – which also have important things to offer.

## Preventive QC

- Preventive QC is checking on the quality _before  _the transaction is completed. Use Preventive QC when:
- The consequences of errors are high, or
- The error is not recoverable.
- Examples:
- Supervisors sign termination letters even though specialists sign other correspondence
- Occupancy supervisor reviews 100% of rent calculations for new staff (for some period of time).

## Preventive QC

- More examples:
- Intake specialist presents a “dry run” of the briefing to staff before doing the real thing in front of participants.
- What examples of preventive QC occur at your PHA?
- Try to identify at least one additional area (related to income and rent) where preventive QC might be helpful.

## Limits of Preventive QC

- Preventive QC can be an important tool but what are its limitations?
- PHA’s don’t have time or staffing to double and triple check every transaction –you need to prioritize.
- Use Common Sense:  Not everything requires this level of effort.

## Periodic Quality Control

- In contrast, periodic quality control happens after the fact.  Work has already been done and reported.
- It’s purpose is to take a “picture-in-time” assessment of PHA operations that will help improve the program and prevent future errors.
- Sound familiar?  Yes – HUD’s RIM reviews and SEMAP audits are kinds of periodic QC.

## Periodic Quality Control

- During a periodic review HUD systematically looks at:
- Policies, procedures, and practices and
- Individual files.
- But you don’t have to wait for HUD. You can also do your own.  You could:
- Look at the same things HUD will be checking, or
- Focus your periodic review differently to learn even more.

## Limits of Periodic QC

- Periodic QC can provide an accurate point-in-time assessment, but:
- It doesn’t help managers see trends
- Results come infrequently so it doesn’t help managers take corrective action during the year.
- The level of effort required may seem overwhelming (but HUD’s RIM Review guide provides a good model for the review).

## Continuous Quality Control

- Continuous QC is used to help with day-to-day management.
- It involves analyzing a few (but very important) factors on a regular, continuing basis.  For example:
- Tracking on-time reexaminations each month
- Looking at accuracy rates by individual or transaction

## Continuous QC

- Continuous QC helps managers:
- Focus more quickly on vulnerable areas.
- Look at trends, rather than point-in-time information.
- HUD does some continuous QC of 50058 submissions and uses the data to:
- Send error reports to PHAs, and
- Assess in general terms how PHAs are doing.
- But only the PHA can implement an effective continuous QC program.

## Limits of Continuous QC

- Continuous QC is a powerful tool.
- But, remember, the scope of the Continuous QC is limited.
- This means:
- Continuous QC provides only a limited picture (hopefully of the key areas)
- Continuous QC should be supplemented by Periodic Reviews that take a broader view.

## Continuous QC Example

- Page 6-11 and 6-12
- Look at the “Error Analysis” chart
- QC results are shown by quarter & individual
- Caseload = annual caseload
- Errors = things HUD would call findings
- Comments = errors that would not be findings
- Types of errors are defined on page 6-12

## Continuous QC Example

- What issues does this report raise for you?
- What are your concerns about:
- Productivity?
- Types of errors?
- Error Rates?
- What are the implications of the  information you see for this PHA?

**Notes:**
NOTE:  This is the only slide that uses the drop down bullet by bullet.  The reason for this is that I’m trying to give you some way to remember to tell people that the answers on page 6-51 and 6-52 but I don’t want them to know about it ahead of time.  Do you want to give yourself this reminder some other way>

## Chapter 6.  Section 3.Designing a QC Program

## Designing a QC Program

- Now that we have discussed each type, it’s time to think about how to design your program.
- Before you begin it is very important to think through:
- What you want your QC program to look at.
- What you will do with the information.

## Designing a QC ProgramWhere to Start

- To focus your program ask three questions:
- What are the outcomes you want to achieve?
- What would success look like for each outcome?
- What is the best way to confirm (track, measure) performance in each area?
- Then decide:
- Which measures are useful and feasible.
- How you are going to record the information.
- Who needs to know the results.

## A Simple Example- Sleep

- What is my desired outcome?
- A good night’s sleep
- What would success look like?
- Eight hours of uninterrupted sleep
- What is the best way to measure?
- An independent 3rd party could observe my sleep.
- I could track myself the time I go to bed, time I wake up, and number of interruptions.

## Simple Example (Cont)

- Decision:
- Having someone watch me sleep could be expensive and disruptive
- Instead, I will track sleep patterns myself
- I’ll write this down in a notebook
- No one needs to see it but me (unless I think maybe my doctor should know)

## A More Serious Example – Utility Allowances

- What is my desired outcome?
- Current and correct utility allowances will always be used in rent calculations.
- What would success look like?
- Utility allowances would be used correctly in 100% of the files.
- Are there other ways you would describe success re: utility allowances?

**Notes:**
Trainer Notes: 
WHAT happens is distinct from WHY it happens. 
Understanding WHAT and WHY both require conversations with staff. 
The analyst must probe until it is possible to answer both questions.
RIM may be a new process but your oversight of staff work is not new to most of you. What have you found are the most likely “why” reasons?

Sample discussion for Computerized Tools -- Do staff know how to use the programs? Do we have adequate computers to run the programs (enough computers, with fast enough speeds and large enough memories, that don’t go down when they are needed?) Do I know for sure that my software does the calculations correctly?
Sample discussion for computerized aids -- These might include forms, checklists, interview scripts, form letters, record-keeping formats) Are they clear, simple to use, and readily available to staff?Do we have them for all activities that they could help streamline?
Communication (need to tell staff how to do it and why to do it)
Staff capacity refers to (training, knowledge, skills and abilities, attitude)
Facilities refers to meeting rooms, fax machines, etc.
Supervision is a source of error when it fails to correct on-going problems, or directs staff to take incorrect actions.
Organizational culture is a sources of error when the incentives/attitudes around the organization encourage incorrect behavior
Policies and procedures are a source of error when they are unclear or incomplete.

## A More Serious Example – Utility Allowances

- What are possible ways to measure the outcome?
- 100% file review in advance (preventive)
- 100% file review after the fact
- Sample file reviews after the fact
- What else could we do to see if the correct utility allowances are being used consistently?

**Notes:**
Trainer Notes: 
WHAT happens is distinct from WHY it happens. 
Understanding WHAT and WHY both require conversations with staff. 
The analyst must probe until it is possible to answer both questions.
RIM may be a new process but your oversight of staff work is not new to most of you. What have you found are the most likely “why” reasons?

Sample discussion for Computerized Tools -- Do staff know how to use the programs? Do we have adequate computers to run the programs (enough computers, with fast enough speeds and large enough memories, that don’t go down when they are needed?) Do I know for sure that my software does the calculations correctly?
Sample discussion for computerized aids -- These might include forms, checklists, interview scripts, form letters, record-keeping formats) Are they clear, simple to use, and readily available to staff?Do we have them for all activities that they could help streamline?
Communication (need to tell staff how to do it and why to do it)
Staff capacity refers to (training, knowledge, skills and abilities, attitude)
Facilities refers to meeting rooms, fax machines, etc.
Supervision is a source of error when it fails to correct on-going problems, or directs staff to take incorrect actions.
Organizational culture is a sources of error when the incentives/attitudes around the organization encourage incorrect behavior
Policies and procedures are a source of error when they are unclear or incomplete.

## A More Serious Example

- Decision:
- 100% review would be too time consuming (before or after)
- Therefore, review a sample of files (after the fact) to see how often staff have used the correct utility allowance
- Generate a report that summarizes errors by employee, by program type, by transaction type.

**Notes:**
Capping Deduction
	The stated cause is that people are confused, but more discussion reveals that there are some differences in this regard between new staff and more experienced staff

## Thinking About Outcomes

- Choosing the outcomes you want to use is an important part of the design process.
- Outcomes can be expressed in very general or very specific terms.
- Some “big picture” outcomes might include:
- Consistency in data collection, documentation
- Complete and accurate verifications
- Accuracy in rent calculations
- Timeliness
- Complete, accurate reporting

## Thinking About Outcomes 

- Even broader outcomes might include:
- No findings on the next RIM review
- Informed decision making by applicants, participants and HCV landlords
- Fair and equitable treatment
- Equitable workload
- Being a SEMAP high performer
- Positive perceptions – we’re a good agency.

## Thinking About Outcomes

- Caution:  Generally more specific outcomes are better understood and easier to measure.  For example:
- Reduce departmental error rate to less than one percent
- Complete 100% of annual reexaminations accurately and on-time
- The more specific the outcome, the easier it is to track performance.

## Designing a Program to Support Outcomes

- Page 6-20 through 6-21 show examples of turning desired outcomes into a QC plan.
- Follow along on the chart
- We will walk through Row 1 and Row 2 together

## The End Result

- The end result of this thinking will be a Quality Control Plan.  (see page 6-25).
- The planning itself is more important that the format for the plan.
- Think through the process before you begin data collection
- Collect information about the items that have the most effect on error reduction
- Be realistic about what data you can collect.

## What’s Next?

- The balance of this chapter discusses 5 important “how to” aspects of QC:
- Conducting file reviews
- Conducting a “general” review of operations
- Using reports to track quality
- Addressing staff skills and skill gaps
- Communicating about error reduction and quality control

## Chapter 6.  Section 4.How to Aspects of Quality Control

## Conducting File Reviews

- Files reviews are used:
- To determine accuracy and completeness
- To identify potential policy and management issues
- “Files” for this purpose include the 50058 record as well as paper files.
- Caution: A file review can tell you what has happened, but not necessarily why.

## Conducting Tenant File Reviews

- Sample size and characteristics of files selected  depend upon the purpose.
- Generally PHAs use a “directed” sample.
- Directed means selecting the files that contain the characteristics you want to monitor.
- Page 6-28 shows characteristics that may be important to consider for the sample.
- Not every characteristic must be represented each time you review files.

## Conducting File Reviews

**For Continuous QC**
- Purpose is timely feedback on few issues.
- File Selection Priorities would be:
- Each specialist
- Basic transaction types
- Files completed over time

## Conducting File Reviews

**For Periodic QC**
- Purpose is a more comprehensive review.
- File Section Priorities:
- Each specialist (but perhaps more files)
- More transactions types
- Larger cross section of income sources and household types

## Conducting File Reviews

- Tools for Selecting Files:
- PIC reports can generate lists for some transactions.  (See Appendix A, A-25)
- PHA vendor software may be able to generate reports that list files in categories.
- How many files do I need to review?
- QC sample should be sufficient to meet SEMAP requirements

## SEMAP File Sampling

- Useful guideline for determining number of files to review

| Universe | Minimum Sample Size |
| --- | --- |
| 50 or less | 5 |
| 51-600 | 5 plus 1 for each 50 (or part of 50) over 50 |
| 601-2000 | 16 plus 1 for each 100 (or part of 100) over 600 |
| Over 2000 | 30 plus 1 for each 200 (or part of 200) over 2000 |

## Conducting File Reviews

- Logistics
- Use a file checklist to document your review. (RIM checklists are a good start.)
- Complete 50058 calculations without looking at the completed calculations in the files.
- Look for consistency between the paper and electronic files.

## What The Reviews May Show

- Key RIM Review programmatic findings:
- Earned income/pension/public assistance calculations
- Incomplete/outdated PHA policies
- Data not being transmitted to PIC
- Lack of verification of income and deductions
- File data doesn’t match data used to calculate rent
- Improper calculation of utility allowances

**Notes:**
Trainers: --If by this time you have in other Chapters belabored this point  -- just ackonowledge this and pass by.

## What the Reviews May Show

- RIM reviews also found errors related to automation.
- In general, these errors include:
- Programming problems(software vendor)
- Errors related to the way PHAs use automated systems such as:
- Not attending to error reports and
- Inadequate training for staff on the system
- See page 6-32 for details

## Conducting the “General” Portion of a Periodic Review

- Periodic Reviews enable PHAs to take the same broad look that HUD takes during a RIM Review.
- The agenda:
- Look at policy and procedural documents
- Conduct an expanded file review
- HUD’s RIM checklists are good models.

| C. Admin Plans | H.  Rent and HAP |
| --- | --- |
| D. Payment Standards | I. Reexaminations |
| E. Applications | J.  HUD 50058 |
| F. Income |  |

## Conducting the “General” Portion of the Review

- You may need to add to HUD’s checklists:
- Questions covering PHA-specific policies or performance standards
- Protocols for checking how PHA-specific software handles income, rent and subsidy determinations.

## Using Reports to Track Quality

- Perspective is important.
- Identifying individual errors is important but it’s only one aspect of QC.
- Looking at trends (performance over time) provides important insights that lead to error reduction.
- Looking at trends can also help set performance standards.

## Using QC Reports to Track Quality

- What should be tracked regularly?
- How often things happen
- How long things take
- Whether improvement is being made
- How individuals are doing.
- Caution:  Tracking reports can raise flags but may not answer the question: “Why?”

## Analyzing Data

- Pages 6-36 – 6-38.
- Study the three reports
- On-time reexaminations
- Transactions by type
- Error rate trends
- What flags does each report raise?
- What additional information would you like to have?

## Addressing Staff Capacity Issues

- Error reduction will not happen without staff capacity.
- No one is born knowing the rules of assisted housing!  Most PHAs must grow their own “experts.”
- Staff development is a joint responsibility of the organization and the individual.
- Capacity building begins with a mutual understanding of both the job requirements and current performance.

## Addressing Staff Capacity

- A good QC program can support staff development efforts if it includes an analysis of:
- The knowledge and abilities that each job requires
- The actual knowledge and abilities of the staff.
- And, if the PHA uses this information to create:
- A strategy for addressing any gaps in capacity
- A plan for developing staff and monitoring staff progress in meeting performance goals.

## Addressing Staff Capacity

***Skills Set*** is the term used to describe what people need to _know_, _understand_, and _be able to do_.
- See pages 6-40 & 6-41 for a list of the skill sets required by staff who perform income and rent determinations.
- Review the list briefly.  Do you think these items represent the skills needed by staff at your PHA?

## Addressing Staff Capacity

- How to use your analysis of skills sets:
- Your analysis can improve your recruiting efforts and hiring decisions
- Be aware of skill gaps when considering changes in staff roles or position
- Use your understanding of required skills vs. available staff skills to help diagnose performance problems
- Use your understanding of skill sets to select the right employees and to develop employee training plans.

## Addressing Staff Capacity

- Consider using Individual Development Plans to:
- Help set expectations
- Demonstrate a commitment to staff development
- See page 6-49 for a sample IDP.  Note how it sets expectations and specifies development activities that will help the employee succeed.
- Goal #2: Employee will become proficient....
- Goal #2, Activity 3. Attend a 3-day occupancy training program.

## Communicating About Error Reduction and QC

- The Challenge:
- Reducing errors requires staff commitment.
- A QC program must be seen as a positive activity to staff as well as management.
- To succeed, staff need to know:
- What good performance is
- How it will be measured
- How they are doing as performers.

## Communicating About Error Reduction and QC

- Performance expectations help staff understand what is required.  They generally relate to:
- Quality and Quantity of Work
- Timeliness
- Customer Service
- Quality control program helps managers to:
- Set expectations
- Share information about actual performance in comparison to goals
- Recognize and reward good and improving performance

## Communicating About Error Reduction and QC

- Sharing information about performance helps staff keep focused on:
- What is important
- What needs to be done
- What they have accomplished
- Most organizations have too many reports and not enough information.
- Finding effective ways to share this information is challenging.

## Communicating About Error Reduction and QC

- Considering using “Scorecards” to help staff maintain perspective
- Keep them simple (see page 6-45) and focus on:
- Goals and. Progress
- Information that makes paperwork real (household demographics, neighborhoods)
- Quality and quantity of work
- Progress over time
- Accomplishments – things to brag about

## Communicating About Error Reduction and QC

- Scorecards for individuals also can help reinforce expectations and give feedback to staff.

## QC Recap

- A good quality control system should:
- Be comprehensive but not burdensome
- Cover key performance areas
- Fairly and accurately measure performance
- Be perceived as fair by staff and managers
- Few organizations are perfect. QC will find problems.
- But a good QC program will also show the pathway to improvements.

## QC Recap

- The next session talks about how QC data can help you to develop strategies for:
- Responding to RIM findings
- Solving performance problems, and
- Reducing errors.