United States 

Department of 

Agriculture 

Food and 

Nutrition 

Service 

Office of 

Analysis and 

Evaluation 

FINAL SUMMARY REPORT 

March 1998 

Authors: 

Christopher Logan 

Julie Masker 

John Blomquist 

Submitted by: 

Abt Associates Inc. 

55 Wheeler Street 

Cambridge, MA 02138 

Project Director: Christopher Logan 

Submitted to: 

Office of Analysis and Evaluation 

USDA Food and Nutrition Service 

3101 Park Center Drive, Rm. 214 

Alexandria, VA 22302 

Project Officer: Ken Offerman 

This study was conducted under Contract No. 53-3198-6-016 with the Food and Nutrition Service, United States Department 

of Agriculture, under the authority of the Food Stamp Act of 1977, as amended. Points of view or opinions stated in this 

report do not necessarily represent the official position of the Food and Nutrition Service. 

Retailer Pre-Authorization Visit 

Demonstration 
 1 

RETAILER PRE-AUTHORIZATION VISIT DEMONSTRATION

FINAL SUMMARY REPORT

INTRODUCTION

 The Food and Nutrition Service (FNS) of the U.S. Department of Agriculture administers 

the Food Stamp Program (FSP), the nation's largest nutritional assistance program. The purpose of 

the FSP is to improve the food-purchasing power of financially needy households. The program 

provides low-income households with benefits in the form of food stamp coupons or electronicallyencoded cards that enable recipients to purchase eligible food items at authorized retail food stores. 

 Background. In order to be eligible to accept food stamp benefits, retailers must meet the 

eligibility criteria established by law, regulations, and policy, which are administered by FNS. 

According to the current legislation governing the FSP, one of the requirements for a store to be 

eligible is that it: 

 "sells food for home preparation and consumption and (A) offers for sale, on a 

continuous basis, a variety of foods in each of the four categories of staple foods, 

as specified in subsection (u)(1), including perishable foods in at least two of the 

categories; or (B) has over 50 percent of the total sales of the establishment or 

route in staple foods."1

 Retailers interested in participating in the FSP for the first time must apply to their FNS 

Field Office and be approved before participating in the FSP. FNS periodically reëvaluates the 

eligibility of participating FSP stores, typically every three to five years. Field Office staff review 

the application information, follow up with a phone call if necessary and, in some cases, pay an inperson visit to the store. Due to limited resources, however, it has become increasingly difficult for 

Field Office staff to visit stores. 

 In order to prevent ineligible retailers from participating in the FSP, the USDA budget for 

FY 1997 includes funds to support FSP retailer authorization site visits. This money will be used 

primarily to fund store visits by contractors, who will provide the information gathered to the FNS 

Field Office staff making the eligibility decisions.2

 

 1 Public Law 102-225, section 201. Subsection (u)(1) defines staple foods as including meat, poultry, or fish; 

bread or cereal; fruit or vegetables; and dairy products. 

 2

 FNS issued a Request for Proposals (RFP FNS 97-003LMW) to procure these services on February 14, 1997. 
Final Summary Report

 2 

 In the spring of 1996, with the expectation of the FY 1997 funding to support preauthorization store visits, FNS set out to determine the best way to manage these newly-acquired 

resources. FNS also wanted to determine how to ensure that contractors provide sufficient data to 

enable Field Offices to make sound, sustainable determinations of program eligibility. Therefore, 

FNS conducted a large-scale, four-month-long demonstration, the Food Retailer Pre-Authorization 

Visit Demonstration, in which contractor representatives visited over 7,000 stores on behalf of FNS. 

 Evaluation Objectives and Sources. This report summarizes the findings of an FNSsponsored evaluation of the demonstration. In a competitive procurement, FNS selected Abt 

Associates Inc. as the evaluator for the demonstration.3

 The primary goal of the evaluation was to 

determine how best to manage and allocate the additional resources that FNS expected in FY 1997 

for contractor-conducted pre-authorization store visits. 

 In order to answer the research questions posed by FNS, Abt Associates Inc. conducted 

two rounds of on-site data collection activities between July 15 and September 30, 1996. Evaluation 

staff interviewed FNS staff at the Field and Regional Offices, and contractor personnel at the 

corporate and field levels. In addition, evaluation staff observed store visits conducted by FNS Field 

Office staff and by contractor personnel. 

 Special tracking forms created for the demonstration and other FNS administrative records 

were collected from the 16 Field Offices that participated in the demonstration. The database from 

these sources was used to analyze the following: numbers of new applications and reauthorization 

applications for which the subject stores were visited by contractors or FNS; timeliness and 

usefulness of contractor reports; FNS actions to gather additional data from contractor-visited stores; 

and outcomes of the application/reauthorization process (i.e., approval, denial, or withdrawal). 

 Organization of the Report. The remainder of this report is organized in three parts. 

First, the report describes how FNS organized and operated the demonstration. Second, the results 

of the demonstration are presented, including findings based on the interviews with FNS and 

contractors, the tracking forms, contractor cost information, and the documentation of appeals of 

decisions to the FNS Administrative Review Branch. The final section presents the conclusions 

from the evaluation, combining the insights of FNS and contractor representatives with those of the 

 

 3 USDA Food and Nutrition Service Contract No. 53-3198-6-016. 
Final Summary Report

 3 

evaluation team. Appendix A presents supplemental information on the sampling and analysis of 

the tracking forms. 

DESCRIPTION OF THE DEMONSTRATION

 The demonstration involved the efforts of various staff at FNS headquarters, Regional 

Offices, and Field Offices. Headquarters staff provided contract support and guidance. Each 

Regional Office designated a contracting officer and a contracting officer's representative (COR), 

with backup for the COR. Sixteen Field Offices, spread over all seven FNS regions across the 

nation, participated in the demonstration. Taken together, these Field Offices were responsible for 

nearly one-third of all authorized retailers in the nation. The sites included three large Field Offices, 

with over 5,000 retailers each; eight medium-sized offices, with 3,000 to 5,000 retailers each; and 

five small offices, with fewer than 3,000 retailers each. Through competitive procurements, the 

Regional Offices contracted with private firms to work in the territories covered by these Field 

Offices. 

 Demonstration planning activities began in late 1995, culminating in the release of a 

Statement of Work (SOW) by each Regional Office in April 1996. After the contractors were 

selected, the Regional and Field Offices trained contractor representatives in the demonstration 

procedures. The contractors began visiting stores in June 1996. 

 Nine firms were awarded contracts and participated in the demonstration. Six of the 

contractors served two or more sites. The contractors' fees, based on their competitive bids in 

response to the regional SOWs, ranged from $25 to $125 per visit. In most sites multiple contracts 

were awarded, but visits were assigned to the contractor with the lowest bid for the site unless that 

firm failed to perform as required. 

 Each of the 16 Field Offices ordered contractor store visits for both new authorizations and 

reauthorizations on an as-needed basis. Field Offices had the flexibility to establish the criteria for 

which stores were assigned to the contractors, based on type of store, geographic location, and other 

considerations. The variety of visits ordered ranged from one site's special focus on stores with 

patterns of excessive FSP redemptions to sites where most stores applying for authorization or 

reauthorization were visited, occasionally including supermarkets. Most Field Offices issued 

separate orders for new authorizations, which had deadlines for contractor reports of 7 to 10 days, 

and for reauthorizations, which had contractor deadlines of 10 to 30 days. (The contractor deadlines 

varied from region to region, depending on the terms of the region's SOW.) Most Field Offices 
Final Summary Report

 4 

geographically grouped the stores in their orders to some degree, in order to reduce the contractors' 

travel time, but low numbers and geographic dispersion of the available stores to be visited limited 

or precluded geographic clustering in several sites. During the demonstration all but four Field 

Offices continued to send their own staff to visit some stores, most often when stores required 

special attention (e.g., because of known concerns about the store's eligibility). On rare occasions 

FNS staff made followup visits when the information from contractor visits was inconsistent or 

inconclusive. 

 Contractors made in-person store visits in which, after receiving the consent of a store 

representative, they completed a checklist of the food inventory and took photographs (including the 

staple food stock, the check-out area, and the store exterior). The procedures for the inventory 

checklist and photographs were modeled in part on approaches used by FNS staff, but the 

demonstration contractors took substantially more photographs (at least 12 per store) than has been 

the practice of FNS. (FNS staff typically took photographs only to document stores' deficiencies, 

but contractors were expected to photograph foods from all four groups of staple foods regardless of 

how well-stocked the store was in each food group.) In the Western Region, contractors were also 

required to sketch the store's layout. The contractors then submitted reports, comprising a cover 

page, the checklist, the photographs, and (in some regions) supplemental information, to the FNS 

Field Offices. Contractors spent an average of 95 minutes per store on preparations, conducting the 

visit, and producing the report; the typical in-store time was 15-25 minutes. 

 The FNS Field Offices reviewed the information provided by the contractors, both to 

determine the acceptability of the contractor reports and to assess what the reports indicated about 

the subject stores' eligibility. Most Field Offices conducted separate quality control checks on the 

contractors' reports before using them to determine the stores' eligibility. The most common 

problems detected by these reviews were missing or unclear photographs and inconsistencies 

between the checklist and the photographs. Most Field Offices provided prompt feedback to the 

contractors regarding deficiencies in the reports, either in writing, by telephone, or both. 

 The Field Offices were almost always able to decide retailer eligibility using the 

contractor reports. FNS made very few follow-up visits to stores visited by the contractors. About 

half of the Field Offices viewed the photographs and the checklists as equally important to their 

eligibility decisions, but the other half rated the photographs as more important. A major challenge 

was the need to review each store on its own merits, because the FSP's Criterion A for retailer 
Final Summary Report

 5 

eligibility (ample variety of staple foods) does not provide quantitative standards for staple food 

inventory. An additional challenge was the need to assess the store's potential eligibility under 

Criterion B (over 50 percent of total sales in staple foods). 

DEMONSTRATION RESULTS

 Basic Statistics. When the authority to commit demonstration funds expired on September 

30, 1996, the Field Offices had ordered a total of 7,232 contractor visits, as shown in Exhibit 1. The 

number of visits ordered ranged from 173 in the smallest site (Wichita) to 1,111 in the largest (Los 

Angeles). The average cost across all sites and visits was $70 per visit; the average cost in 

individual sites (reflecting combinations of two or more contractors in the sites that had turnover) 

ranged from $30 in Boston to $118 in New York City.4

 

 4 The range of the final average cost differs from the $25 to $125 range of bids by participating contractors, 

because each of the sites with the lowest and highest bids (Detroit and New York, respectively) used two or more 

contractors with different fees over the course of the demonstration. 
Final Summary Report

 6 

Exhibit 1 

ACTUAL CONTRACTOR VISITS ORDERED BY SITE

Region/Field Office Total Number of 

Retailersa

Store Visits 

Ordered 

Total Spendingb

 Cost per Visitb

Mid-Atlantic Region 

Trenton 4,010 568 $40,179 $71

Harrisburg 3,138 243 20,431 84

Northeast Region 

New York City 9,788 974 115,406 118

Wallingford 1,748 236 8,140 34

Boston 3,782 297 8,901 30

Southeast Region 

Tampa 4,200 349 11,639 33

Raleigh 6,442 383 35,079 92

Midwest Region 

Detroit 3,618 407 35,827 88

Grand Rapids 3,361 525 31,284 60

Southwest Region 

Austin 4,887 727 39,985 55

Little Rock 2,461 522 29,993 57

Mountain Plains Region 

Denver 1,788 236 16,237 69

Wichita 1,393 173 11,929 69

Western Region 

Los Angeles 8,497 1,111 65,175 59

Sacramento 3,418 291 23,037 79

Phoenix 2,185 190 13,343 70

 

TOTALS 64,716 7,232 $506,585 $70

a

 Retailer counts as reported in the SOW for the demonstration. 

b

 Cost per visit represents the average fee for all visits as ordered. Final cost per visit may have been lower to the extent that 

penalties for late deliverables were assessed. 
Final Summary Report

 7 

 Field Office Experiences. All but three of the Field Offices experienced significant 

problems with contractor performance at some point in the demonstration, but most of the 

difficulties were in the early months. The principal problems were late, incomplete, inconsistent, or 

poorly-presented reports; refusal of work assignments was a less common problem that was 

important in a few sites. Nearly all of the early performance problems were resolved by the fourth 

(and last) month of the demonstration, either through improved effort by the original contractors or 

by replacing them with better-performing (albeit more expensive) contractors. A substantial 

minority of the Field Offices (38 percent) had to change contractors at least once. Sixty-one percent 

of the Field Offices gave ratings of "moderately high" or "high" satisfaction to the contractors they 

were using near the end of the demonstration. Seventeen percent rated their level of satisfaction as 

"moderate," and 22 percent said that their level of satisfaction was "low" to "moderately low." 

 Most Field Offices felt that the additional steps in the retailer authorization and 

reauthorization process during the demonstration affected how they allocated staff time. Some 

offices spent more time overall on retailer management activities, relative to other Field Office 

duties.5

 Others reported reductions in the proportion of their retailer management time devoted to 

monitoring retailer participation (such as reviewing redemption reports). Some offices reported both 

of these effects. 

 Field Office staff shared several widespread views of the demonstration experience: 

 · Field Office staff saw the demonstration as reducing FNS time devoted to store visits, 

but increasing the overall number of visits conducted. 

 · Sites that experienced significant problems with contractor performance stressed the 

added workload associated with correcting this situation. 

 · The compressed schedule of the demonstration and the record-keeping for the 

evaluation also contributed to the perceived workload effects. 

 · Contractors were seen by Field Office staff as an important resource, particularly for 

covering territories with sparse, far-flung retailer populations. 

 · Field Office staff believed, however, that contractor visits are, in some ways, less useful 

 

 5 The general category of FSP retailer management includes the following activities: authorization and 

reauthorization of retailers to participate; providing training and technical assistance to retailers regarding FSP 

rules and procedures; monitoring retailers' patterns of redemption activity to detect store closures and possible 

program violations; and sanctioning retailers for program violations. 
Final Summary Report

 8 

than FNS staff visits. FNS staff can answer retailer questions, spot signs of fraudulent 

behavior, verify ownership, and make preliminary authorization decisions on the spot. 

 Nearly two-thirds of the Field Offices substantially underspent their allocations for the 

demonstration (i.e., ordered two-thirds or fewer of the number of visits projected at the outset, based 

on the available funds and contractor fees). The reasons for the shortfall included Field Office staff 

shortages, competing priorities (such as the FNS "sweep" of a large number of stores in one site), 

and lost time due to problems with contractor performance. The shortfall was partially offset by the 

four Field Offices that ordered more than the originally-projected number of visits, because of 

lower-than-expected contractor costs or availability of reallocated funds from the sites that 

underspent their allocations. 

 Outcomes of Contractor and FNS Store Visits. As part of the evaluation, Field Office 

staff completed Initial Tracking Sheets for all new authorizations and reauthorizations processed 

during the demonstration, and Follow-Up Action Reports on all contractor-visited stores. The 

evaluators analyzed the store visit tracking data contained in a sample of these forms completed 

during the demonstration period. The sample is composed of 2,194 store visits, 1,934 of which were 

conducted by contractors, with the remaining 260 visits conducted by FNS. The sample includes 

data from all 16 demonstration Field Offices and nine contractors.6

 Overall, the tracking data indicate that contractors successfully conducted retailer visits 

on behalf of FNS. Nearly 99 percent of visits were completed. Exhibit 2 summarizes the outcomes 

of the completed contractor visits, overall and by whether the visit was for an initial authorization or 

reauthorization. Work was delivered to FNS on a timely basis in 90 percent of the visits. 

Information from contractor reports contributed to or supported authorization or reauthorization 

decisions in 96 percent of cases (i.e., without a follow-up contact visit by FNS).

 

 6 For further information on the sample and weighting methodology, see Appendix A. 
Final Summary Report

 9 

 There is considerable variability in outcomes, however, by type of visit. The second 

column of Exhibit 2 shows that contractor work was delivered on a timely basis more frequently for 

reauthorization visits than for new authorization visits—92 percent vs. 81 percent. This statistically 

significant difference7

 is not altogether surprising, because the time allotted by FNS for delivery of 

new authorization visit reports was almost always less than for reauthorizations, and new authorization visits tended to be more geographically dispersed than reauthorization visits because they could 

not be grouped in batches to ease scheduling. Both characteristics made it more difficult for 

contractors to turn in new authorization deliverables quickly and efficiently. 

 

 7 Throughout this report we say that a difference between two means estimated from sample data is 

"statistically significant" if there is a 10 percent or smaller chance that the true difference in the population is 

actually zero. 

Exhibit 2 

OUTCOMES OF CONTRACTOR STORE VISITS, BY TYPE OF VISIT

Visit Type Work 

Delivered 

on Timely 

Basis 

Requested 

More 

Information 

from 

Contractor 

Authorization 

Decision Based 

on Visit 

Store 

Approved or 

Reauthorized 

Store Denied 

or Withdrawn

All contractor visits 89.9% 3.7% 95.7% 92.2% 7.8%

New authorizations 81.0a 6.8a 95.3 87.2a 12.8a

Reauthorizations 92.3a 2.8a 95.8 93.7a 6.3a

SOURCE: Initial Tracking Sheets and Follow-up Action Reports for visits in the analysis sample. 

a

 Difference between new authorizations and reauthorizations significant at the 0.10 level. 
Final Summary Report

 10 

 The fifth column in Exhibit 2 indicates that store reauthorization rates were over six 

percentage points higher than approval rates for new stores, a statistically significant difference. 

Also, as displayed in the third column, FNS was more than twice as likely to request that contractors 

provide additional information or repeat visits to stores following new authorization visits than 

following reauthorization visits (in 7 percent vs. 3 percent of cases, again a statistically significant 

difference). FNS made such requests, as authorized under the SOW, when the contractors failed to 

provide complete and consistent information. The results are consistent with evidence from the 

process interviews that Field Offices subjected new authorizations to a higher level of scrutiny than 

reauthorizations, posing a greater need for complete and reliable data on the store in question. Stores 

that are applying for reauthorization have already passed through the initial authorization process 

and require less verification. FNS was no less likely, however, to be able to make an authorization 

decision using information from the contractor visit in new authorization cases than in 

reauthorization cases. 

 The outcomes of FNS store visits, by type of visit, are compared to those of contractor 

visits in Exhibit 3. It should be noted that the sample of FNS visits is much smaller than the sample 

of contractor visits (260 FNS visits versus 1,934 contractor visits). With this in mind, the sample 

indicates that FNS visits also led to high rates of authorization decisions, both for new store 

applications and for reauthorizations, and high rates of approval or reauthorization. Over 97 percent 

of FNS visits provided sufficient basis for an authorization decision, and just over 95 percent of FNS 

visits resulted in approval or reauthorization. The rate of authorization decisions based on FNS 

visits was slightly higher among new authorizations than reauthorizations, whereas the 

approval/reauthorization rate was higher among reauthorizations. Neither difference, however, is 

statistically significant. 
Final Summary Report

 11 

 Exhibit 3 allows direct comparison of the outcomes of visits conducted by contractors with 

those conducted by FNS. As shown in the last two columns of the exhibit, overall 

approval/reauthorization rates were slightly higher among FNS-visited stores (95 percent) than 

among contractor-visited stores (92 percent). In addition, FNS had a slightly higher proportion of its 

visits that directly resulted in authorization decisions (97 percent for FNS visits versus 96 percent for 

contractor visits). Although the differences between contractor and FNS visits on these overall 

outcomes are not statistically significant, a different pattern emerges when new authorizations and 

reauthorizations are examined separately. 

 Among all new authorizations, approval rates were significantly higher among visits 

conducted by FNS (95 percent) compared with contractor visits (87 percent)—a more than 7 

percentage point difference. Among all reauthorizations, there were no significant differences in 

approval rates or in the percentage of authorization decisions based on store visits without followup 

between contractor and FNS visits. Thus, approval rates for new contractor-visited stores were 

significantly lower than for either new FNS-visited stores or contractor-visited stores applying for 

reauthorization. All other approval rates were not significantly different from each other. 

Exhibit 3 

OUTCOMES OF CONTRACTOR AND FNS STORE VISITS BY TYPE OF VISIT

Visit Type Percent of Total Visits Authorization Decision 

Based on Visit 

Store Approved or 

Reauthorized 

Contractor FNS Contractor FNS Contractor FNS 

All 93.2% 6.8% 95.7% 97.2% 92.2% 95.2%

New authorization 84.0 16.0 95.3 97.5 87.2a 94.8a

Reauthorization 96.0 4.0 95.8 96.5 93.7 96.1

SOURCE: Initial Tracking Sheets and Follow-up Action Reports for visits in the analysis sample. 

a

 Difference between contractor and FNS visits significant at the 0.10 level.
Final Summary Report

 12 

 One possible explanation for the lower approval rate for new stores when visited by 

contractors (compared with stores visited by FNS) is that the contractor visits provided a basis for 

more systematic and objective comparisons among stores, thereby making clearer the differences 

between eligible and ineligible stores. As the preceding description shows, the process of deciding 

whether to authorize a new applicant retailer when a contractor visited the store was different in 

several important ways from the process when FNS visited the store. For the contractor-visited 

stores, FNS had access to a comprehensive set of photographs, in addition to the food inventory 

checklists (which also were completed by FNS when its staff made visits). The SOW provided a 

structure to ensure consistency in the format and scope of the information. Because the information 

on the stores was delivered to the Field Offices, multiple FNS staff (particularly for both program 

specialists and the Officers-in-Charge) could review and discuss the stores. The fact that contractors 

visited stores that were too distant for FNS staff to visit may also have contributed to the lower 

approval rate, if more distant stores are for some reason less likely to pass FNS' tests of eligibility. 

 An alternative explanation is that the observed difference in approval rates reflects 

underlying differences in approval rates between Field Offices regardless of who conducts the visit. 

The samples of contractor-visited and FNS-visited new stores were distributed differently across 

Field Offices. The Field Offices that conducted the most visits with their own staff were more likely 

to approve new stores whether the visit was conducted by FNS or a contractor. On the other hand, 

two of the three Field Offices with the most contractor visits had approval rates of 72 percent or less 

for new contractor-visited stores, driving down the average for all new contractor-visited stores. The 

differences in approval rates across Field Offices may reflect differences in the retailer applicant 

pool, or in the relative willingness to risk having decisions reversed in the administrative review 

process, or a combination of these and other factors. 

 The limitations of the demonstration design (especially the non-random way that visits 

were assigned to contractor versus FNS visits) make it impossible to determine conclusively which 

of the preceding reasons contributed most to the difference in approval rates. It is therefore 

uncertain to what extent this result is likely to be replicated when contractor visits are used on a 

larger scale. It is important to note, however, that both contractor and FNS visits presumably 

reduced the number of ineligible retailers entering the FSP, because the applicants that could be 
Final Summary Report

 13 

denied without a visit were generally screened out before stores were selected to be visited. The 

demonstration results suggest that contractor visits can be at least as effective as FNS visits at 

detecting stores that should be denied authorization to accept food stamp benefits. 

 Other findings from the analysis of store visit tracking data include: 

 · Field Offices differed greatly in the mix of type of visit (new vs. reauthorization); 

 · all Field Offices had the majority of visits conducted by contractors; 

 · most FNS visits were conducted at stores applying for new authorization, although 

some variation is evident across Field Offices;8

 and 

 · among Field Offices, the approval rates for contractor-visited stores ranged from two 

sites below 85 percent to five sites with approval rates of 100 percent, and the approval 

rates for new contractor-visited stores ranged from 81 to 100 percent. 

 Administrative Review. The denials of new applications and withdrawals of previously 

authorized stores can, like all adverse actions against retailers, be appealed to the Administrative 

Review Branch of FNS. Upon request by the retailer, the case is assigned to an Administrative 

Review Officer (ARO), who considers the appropriateness of the decision in light of the applicable 

laws, regulations, and policies, the retailer file supplied by the Field Office, and any additional 

information provided by the retailer. The decision of the ARO to sustain or reverse the Field 

Office's action represents FNS' final decision on the case; involuntary withdrawals can, however, be 

appealed to the US District Court for judicial review. 

 Among the decisions based on contractor reports, about one-fifth (21 percent) of the 

denials of new applications and more than one-quarter (28 percent) of the decisions to withdraw 

retailers from the FSP were appealed.9

 Of those denials that were appealed, 63 percent were 

sustained by the Administrative Review Branch, and 37 percent were reversed.10 Thus, 

approximately 8 percent of all denials were reversed. For withdrawals, the story is much the same: 

 

 8 The Field Offices with relatively high proportions of FNS visits conducted at stores applying for 

reauthorization also tended to have more FNS visits of all kinds (relative to the number of contractor visits). 

These offices were more likely than others to assign stores to FNS visits based on location or marginal eligibility. 

 9

 These estimates are based on data supplied by the Field Offices and on exception reports of reversed 

decisions prepared by the Administrative Review Branch. 

 10 Appeals withdrawn by the retailer are included in the count of sustained decisions; actions rescinded by the 

Field Office are included in the reversals. 
Final Summary Report

 14 

60 percent of the withdrawals were sustained by administrative review, whereas 40 percent were 

reversed. When the higher appeal and reversal rates for withdrawals are combined, however, about 

11 percent of all withdrawals were reversed—a modest rate but almost half again as high as the 8 

percent rate for denials. These overall averages reflect the substantial influence of one large Field 

Office with especially high rates of denials, withdrawals, appeals, and reversals. If this Field Office 

is excluded, the reversal rate for all denials drops from 8 to 3 percent, and the reversal rate for all 

withdrawals drops from 11 to 4 percent. 

 Reversal rates for denials and withdrawals based on FNS visits in the demonstration sites 

were much lower, but this difference should not be overemphasized, for two reasons. The 

FNS visits were to a quite select group of stores, and relatively few negative actions arose from these 

visits (leading to a substantial probability that the difference in reversal rates was due to chance). 

FNS tracks the incidence of denials and withdrawals but does not track the rates of appeals and 

reversals for these actions, so there is no baseline against which the demonstration experience can be 

compared. 

 There were three common reasons for reversals of Field Office decisions based on store 

visits by contractors and FNS staff: 

 1. The ARO found that the store, although considered ineligible by the Field Office, met 

the minimum standards set by the most recent FNS policy memorandum (dated January 

31, 1996). This policy memorandum indicates that "prudent judgment" should be used 

in making determinations on marginal stores. This reason was cited in 58 percent of all 

reversed cases examined for this study. Among cases involving contractor visits this 

reason was cited in 81 percent of reversals.11

 2. The Field Office did not provide sufficient documentation for the basis of its 

determination, either in the store's file or in communications with the retailer. A Field 

Office must document the specific grounds for a decision made on the basis of a 

contractor store visit. This reason was cited in 21 percent of all reversals, but in only 

10 percent of cases involving contractor visits. 

 3. The Field Office did not adequately review the retailer for qualification under both 

Criterion A (ample variety of staple foods) and Criterion B (over 50 percent of total 

sales in staple foods), or the ARO disagreed with the Field Office's determination 

regarding eligibility under Criterion B. This reason was cited in 20 percent of all 

 

 11 Exception reports explaining reversals were examined for 112 cases reviewed during the demonstration 

period, of which 21 were identified as involving contractor visits. More than one reason could be cited, so 

percentages in this analysis sum to more than 100. 
Final Summary Report

 15 

reversals but only 10 percent of cases involving contractor visits. 

 Administrative Review Branch representatives indicated that well-prepared, consistent 

contractor reports contributed to the sustaining of Field Office decisions. The lack of photographs 

and checklists from some FNS visits outside the demonstration contributed to the reversal of some of 

the decisions based on those FNS visits. The AROs relied heavily on the photographs provided by 

the contractors, in some cases citing inconsistencies between the inventory checklist and the 

photographs as grounds for reversal. The positive response of the AROs to the use of contractorprovided photographs helped encourage the Field Offices to see photography as a very effective way 

to substantiate eligibility decisions. 

 Contractor Costs. Using data from interviews with contractors and other available 

sources, the evaluation produced independent estimates of contractors' costs to conduct store visits. 

The estimated average total cost per visit incurred by the contractors was about $52, 

substantially below the weighted average fee of $70 per visit paid to the contractors.12 The range of 

the evaluation's estimated contractor costs was from $30 to $93. Field labor costs (including 

preparation for visits, travel time, in-store time, and report preparation) accounted for about 55 

percent of total contractor costs; field non-labor costs (mainly travel expenses) accounted for 20 

percent; and management costs (mainly labor) represented about 25 percent. 

 Across the sites, variations in average travel time and store visit volume were the most 

important influences on the estimated contractor cost. Travel time was the most variable component 

of the field labor cost per visit. Much of the management cost for a given site was a fixed cost 

associated with contracting and other startup activities, so the management cost per visit generally 

declined with visit volume. As a result, contractor costs were lowest in largely urbanized, highvolume sites and highest in sparsely-populated, low-volume sites. Analysis of the effect of different 

assumptions regarding pay rates for field staff confirmed that the pay rate is an influential variable, 

posing a clear trade-off between staff qualifications and costs. 

 

 12 The contractor cost estimate includes wages, fringes, direct expenses and overhead, but not profit. The 

weighted average fee paid to the contractors, as cited earlier in this report, does not account for penalties for late 

or unacceptable reports, for which data were not available. 
Final Summary Report

 16 

CONCLUSIONS

 The demonstration showed that using contractors to visit stores is viable and can work very 

well. This success is highlighted by the following key results: 

 · Field Offices found this resource useful enough to order 85 percent of the originallyprojected number of visits. 

 · By the end of the demonstration, after unsatisfactory contractors had been replaced in 

38 percent of the sites, over three-quarters of the Field Offices were moderately to 

highly satisfied with contractor performance. 

 · Despite the unfamiliar challenges of using contractors' reports, over 60 percent of the 

sites rated the quality of contractor deliverables as good or better; a similar percentage 

of sites rated contractors as generally or consistently on time. 

 · In the tracking analysis sample, 90 percent of contractor deliverables were timely and 

96 percent did not require correction by the contractor.13

 · Contractor visits provide sufficient information in the vast majority of cases: 96 

percent of contractor visits enabled the Field Office to make an authorization or 

reauthorization decision without further information-gathering (other than routine 

documentation). 

 · Contractor visits can and do lead to the exclusion of ineligible stores from the FSP. 

The evaluation data indicate that Field Offices took negative actions (denial or 

withdrawal) against 8 to 9 percent of contractor-visited stores; the estimated total was 

649 stores. 

 · Negative Field Office actions based on contractor visits were, in about 90 percent of 

cases, accepted by the retailer or upheld upon appeal. 

 On several of these indicators, there was considerable variability across sites. Some 

contractors substantially exceeded contractual requirements, whereas others were unable to meet 

those requirements and had to be replaced. A minority of Field Offices exceeded their projected 

number of visits ordered, whereas the majority substantially underutilized this resource. Rates of 

 

 13 The fact that about 60 percent of the Field Offices rated the overall quality and timeliness of contractor 

deliverables as good or better may seem inconsistent with the tracking data indicating that over 90 percent of 

individual contractor deliverables were timely and did not require correction. These two sets of data, however, 

are not comparable. The ratings given by the Field Offices reflect their complex and sometimes subjective 

expectations regarding what contractors should do to meet their needs, above and beyond the minimum standards 

of compliance with the SOW. The tracking data reflect performance only relative to those more objective but 

lower standards for which data were collected. 
Final Summary Report

 17 

approval or reauthorization for individual Field Offices (which are subject to some sampling error) 

varied considerably as a result of the differences in contractor report quality combined with 

variations in approaches to authorization decisions. 

 The contractors in the demonstration proved that they could effectively serve as FNS' 

"eyes" and credibly represent the FSP. Contractors can be a valuable extension of FNS' reach, 

especially in more distant areas. 

 Lessons Learned. Exhibit 4 presents a list of the best practices employed by contractors 

and Field Offices. This list reflects the observations of FNS staff, contractors, and the researchers. 

In each section of the exhibit the best practices begin with startup activities, progress through the 

issuance and completion of call orders, and end with the submission and review of deliverables. 

These practices include approaches, techniques, and tools that contributed to smooth, efficient, and 

effective operations. They are worthy of consideration as FNS proceeds to implement store visit 

contracting on a larger scale, although they may require adaptation to other contractors' or Field 

Offices' circumstances. 
Final Summary Report

 18 

Exhibit 4 

BEST PRACTICES

Best Practices by Contractors

 · Thorough training by contracting organization, which includes a field component 

 · Good working relationship with the Field Office 

 · Automated tracking system 

 · Mapping software that plots store locations and assists in planning travel routes 

 · Photographs that are sufficient in number and content to permit an adequate assessment of the staple 

food stock, as well as overall sense of the store 

 · Methods to correctly match photos with stores, such as using a store identification placard in photos or 

one roll of film per store 

 · Computer software that generates labels for photographs and mailings 

 · In-house quality control process that identifies problems prior to submission of reports to FNS 

 · Neat, professional, flat deliverables that are organized in a consistent fashion 

Best Practices by Field Offices

 · Identification of a store visit coordinator 

 · Adequate training and tools (such as checklist) for staff to ensure consistency in the approach to 

assessing the adequacy of deliverables and making eligibility decisions 

 · Detailed and thorough training of contractors by Field Offices 

 · Communicating clearly the expectations for Field Office needs and contractor performance 

 · Automated process for issuing and tracking call orders 

 · Regular issuance of call orders; steady and adequate flow of work, negotiated by Field Office and 

contractor 

 · Geographic grouping of stores in call orders 

 · Prompt review of contractor deliverables 

 · Ongoing feedback to contractors on the quality of deliverables and issues/problems as they arise 
Final Summary Report

 19 

 FNS has incorporated these and other lessons from the demonstration into the Request for 

Proposals (RFP) for the FY 1997 store visit contracts and other preparations for the future use of 

contractors to conduct store visits. Both FNS and the future contractors will benefit from the 

following advances: 

 · Better information about FNS' plans and expectations for contractor store visits. 

The FY 1997 RFP contained much more information on the timing and location of 

planned store visits, and on their likely distribution between new authorizations and 

reauthorizations, than was available to the demonstration contractors. This information 

should help ensure that contractors are better prepared to meet FNS' needs. 

 · More emphasis on training for reviewers employed by contractors. As part of the 

demonstration evaluation, a guidebook for contractors has been produced, explaining 

how to conduct visits and providing samples of photographs, survey forms, checklists, 

and store sketches. Training for the FY 1997 contractors will be more formalized, 

including sessions at the national and regional or field level. To remove disincentives 

to training, contractors will be paid separately for training costs, rather than having to 

recover those costs from fees for conducting store visits. 

 · Changes in rules and procedures for ordering visits to increase efficiency. Field 

Offices will be able to issue larger orders for store visits, both to reduce ordering time 

and to give contractors a better flow of work. The plans for FY 1997 put more 

emphasis on reauthorization visits, which can be conducted more efficiently through 

geographic clustering and advance scheduling. 

 · Changes in rules and procedures for conducting visits to elicit more consistent 

contractor performance. The FY 1997 RFP spells out more clearly the expectations 

for contractor deliverables, including a specific requirement for at least 17 photographs. 

 Contractors are required to employ field coordinators to facilitate communications 

with the FNS Regional and Field Offices, and to propose and follow clear plans for 

training reviewers. Contractors will be allowed an initial trial period of 30 visits per 

region, after which they will be subject to liquidated damages of up to $200 per 

deliverable. 

 · Changes in the format of deliverables to facilitate use by Field Offices. The staple 

food checklist will be modified so that Field Office staff can more easily determine the 

extent of variety in the store's inventory in each staple food group. A store layout 

sketch, as used in the Western Region during the demonstration, will be used to provide 

better information on the nature of the business and the stock of non-staple foods and 

non-food products. 

 · Realigning responsibilities for administering store visit contracts. To ensure 

consistent contract terms and to better target agency resources, FNS has issued a single 

RFP for the FY 1997 procurement, and the contracts will be administered at the 
Final Summary Report

 20 

national level, with technical support from the Regional Offices. The Field Offices will 

have less responsibility for contract administration and will be able to focus more on 

using contractor visits to accomplish their ultimate objective: screening out ineligible 

stores from the FSP. 
Final Summary Report

APPENDIX A 

TRACKING DATA SAMPLE AND WEIGHTING METHODOLOGY

 During the four months of the demonstration, FNS Field Office staff completed up to two 

evaluation forms for each new applicant store and each store subject to reauthorization. The Initial 

Tracking Sheet (ITS) was completed for all stores, regardless of assignment status. For each store, 

the ITS records tracking information, assignment status (no visit, contractor visit, or FNS visit), and 

(for contractor-visit stores) contractor name and assignment date. For stores visited by FNS, it 

records whether additional information was requested from the retailer, the final outcome (approval 

or denial/withdrawal), and the final action date. For no-visit stores, the final outcome is recorded, 

along with the date of the action.14

 In addition to the ITS, a Follow-up Action Report (FAR) was completed following 

administrative action for all contractor-visited stores. The FAR records, from the viewpoint of FNS 

staff, the outcomes associated with contractor visits, including timeliness of contractor deliverables, 

completeness of contractor information and additional information requested, action taken on 

contractor reports, whether a follow-up FNS visit was conducted, the final administrative action 

taken (approval or denial/withdrawal), and the date of the action. 

 

 14 This analysis does not include data on no-visit stores. 
 Appendix A: Tracking Data Sample and Weighting Methodology

Analysis Sample and Weights

 Because the sample generally represents the entire period of demonstration operation, there 

should be few inherent biases in the data other than inevitable sampling error stemming from the 

random process used to select store visits for analysis. We use weighted frequencies in our analysis 

to make the sample representative of the known universe of visits. Contractor visits are weighted so 

that the weighted sample represents the total contractor visits ordered by each Field Office over the 

period of the demonstration, except in three sites where the weighted sample reflects visits ordered 

through August 16.15 Individual weights for contractor visits were determined separately for new 

and reauthorization visits in each Field Office; the weights were computed as the total number of 

(new or reauthorization) visits ordered, divided by the number of visits in the sample.16 The 

weighted sample is then equal to the total number of new and reauthorization visits ordered. FNS 

visits are weighted to represent the total number of visits in each Field Office for which Abt 

Associates received and batched ITS forms.17

 The effect of these weights on relative sample sizes is shown in Exhibit A-1, where the 

weighted and unweighted samples are compared by Field Office. References to numbers of visits or 

percentages in the report apply to the weighted sample, unless otherwise indicated. In interpreting 

the results, readers are cautioned to recall the size of the unweighted sample, which determines the 

precision of the estimates. The entire sample of 2,194 visits allows fairly precise estimates, but the 

much smaller samples for individual Field Offices yield much less precise estimates. 

 

 15 Delays in receipt of data from these sites limited the analysis to this time period. 

 16 The numbers of new and reauthorization visits ordered by a Field Office were obtained from semi-monthly 

Regional Tracking Reports produced by FNS. 

 17 The actual universe of FNS visits at any exact point during the demonstration is unknown. All 495 ITS 

forms for FNS visits were batched, but only the 260 forms in the sample were processed and used in the analysis. 
 Appendix A: Tracking Data Sample and Weighting Methodology

Exhibit A-1 

UNWEIGHTED AND WEIGHTED SAMPLES, BY FIELD OFFICE

Field Office Total 

Visits in 

Sample 

Percent of All 

Visits 

Weighted Total 

Visits 

Percent of 

Weighted Visits 

Trenton 287 13.1% 568 7.8%

Harrisburg 79 3.6 282 3.9

New York City 303 13.8 1112 15.3

Wallingford 60 2.7 246 3.4

Boston 133 6.1 358 4.9

Tampa 105 4.8 349 4.8

Raleigh 36 1.6 125 1.7

Detroit 112 5.1 422 5.8

Grand Rapids 129 5.9 571 7.9

Austin 114 5.2 619 8.5

Little Rock 127 5.8 646 8.9

Denver 119 5.4 236 3.2

Wichita 62 2.8 179 2.5

Los Angeles 438 20.0 1120 15.4

Sacramento 57 2.6 245 3.4

Phoenix 33 1.5 190 2.6

Total 2194 100% 7268 100%