the experts disparate impact analysis not because the court found flaw with the analysis itself but because the court believed that the factual basis or more particularly part of the factual basis was weak in two thousand six hundred reasoned percs codings if accurate the courts believed that that the data was weak was clearly erroneous as we've discussed in our brief but that this court has often analyzed operate exclusions and held that there is a line when considering the sufficiency of the data over which the district court as gatekeeper should not cross and here let me ask apply and the district court has discretion to determine how best which tools within the dogbert case should be used to assess the evidence right but in assessing them they were misapplied and the court rejected all other evidence which coom home acknowledges there are other ways to prove reliability the district court credited none of the other evidence that we showed the anecdotal that corroboration of expert testimony that what you're referring to and it is undisputed what the experts said is that when he was analyzing and looking at the consistency with which the panelists looked at these photos which made up only part of the race identifications he compared those observations to race self identification that he had for the same persons in all fifty seven cases for he had both and he found a very high rate of consistency ninety five point seven percent when compared with d m v data and eighty percent when compared with kaplan's own higher data reflecting self i d s and the experts undisputed testimony was that was part of the reason and part of the way he review of that is abuse of discretion but here the court erred in two ways that constituted two major ways that constituted an abuse of discretion one was in clearly in a clearly erroneous assessment of the evidence that is crediting none of the evidence that the government showed that tended to show that these photographs the observations were reliable and had hallmarks of reliability and finding in misapplying instead specific factors and finding we didn't satisfy any we did satisfy at least three and perhaps four of the enumerated factors but when the court applied those factors it misapplied them and eroni asli assessed the evidence and that in this case the government met at least well first of all the district court assumed we met testability that is assumed in the opinion so i won't address that well i'm not so sure about that in your let's just ok and so how was that matter here in this case the expert tested in all fifty seven cases where he had two sources that there was a high degree of consistency and he said that constituted a low error rate and that evidence is undisputed so the fifty seven is what supports the test tested in your view it supports the test it also the high degree of into reader agreement that is among the five panelists as to  a very high number of the photos they reached seventy five percent of the photos they reached one hundred percent agreement and keep in mind they were using five categories so the expert used a murder not a very small sample though that  the proof offered to the court with respect to those consistency among those readers but i thought that was a very small sample no it was in all of the cases where photographs were looked at by the panelists if there were eight hundred ninety one photographs looked at as to seventy five percent of those photographs there was perfect agreement as to eighty eight percent of those photographs which was seven hundred eighty eight of them four out of five persons agreed of the palace so that that was the interrelated agreement covers every single photo that they observed the fifty seven cases to i think your honour's talking about the fifty seven cases and that is also all of the instances where we had both sources of identification for the same people so what's the next factor the next factor your honor is we satisfied showing the existence and maintenance of standards and  controls the expert as i said chose five persons who had immediate and of twenty four years of experience in the time he was one of the persons in only fit in only fifteen photos did he and his staff look at them and produce aris identification from the photograph in only fifteen of the eight hundred and ninety one cases sometimes the raiders were given surname sometimes not i gather no  well no every were given every photograph that was given to the panelists had the name of the person depicted in the photograph on the photograph in some way there were none that did not have the name the palace were not instructed to use the names there is no evidence that they did and there is no evidence that any person was in fact identified by last name but it's to the extent that is a factor that introduces some sort testimony here to show that it is our burden to show that the process was reliable however the the supposition that it may have created an undetected bias is is a supposition and in the context of gate keeping analysis because the discretion is so great with the trial court it is important where the trial court passes that line and credits purely supposition are purely hypothetical challenges to enforce that limitation on the discussion is not unfettered here here and we did show that the names were on the photographs so there is not a single person however who there is any evidence that person i was actually identified using the last name and so i was not you and why was the name given to them the name was put on the photos really as a housekeeping administrative matter to make sure that they were all the expert could be sure that each person who was depicted in the photo was in fact the person whose record was delivered by the department of motor vehicles because some of the photos were delivered with the name in the file the electronic file name by the department of motor vehicles so when the photo was printed out the name did not appear so the expert made sure that they were each properly identified as a package any outer you're saying that so we also presented studies cited studies particularly the lambert study out of kansas where there was visual means used to race identify motorists the transients motorists data and that case was not accepted because of the difficulty of identifying persons as they drive by but the portion of the benchmark that was attained by stop data were persons were still they were not in motion and they were observed in their vehicles was accepted we cited that in the district court acknowledged in her order that both parties cited studies where visual identification has been used and in those studies although some of the studies did point out there may be people may encounter difficulties under certain circumstances interbreed or reliability multiple people looking at the same exact image or person was proposed as a means of resolving those difficulties so we also show studies now are they traditional peer review in the sense that the actual data you know that was compiled here was published and reviewed no but our position is that this kind of function of looking at people and being able to discern generally speaking whether they are black or white is so commonly accepted that we do it every day witnesses certainly do it every day that using appear on orders not to do it is not true actually we don't own players not to do it we tell employers to seek self i d first and then if they do not have self id from their employers to use visual identification and that's exactly what we did here in fact at the end of the day after all of the da came in out of the eight hundred eighty persons who were race identified in the applicant pool here one thousand one hundred eighty eight of them almost sixty percent were identified using self identification good morning ma'am please the court my name is gerald mammon of cipher shop and i represent the defendants in this case as your honour's questions to the city council indicated at issue here is whether or not the district court abused it discretion with respect to its treatment of the e e o c is expert report i think there's no dispute that summary judgment rose and fall on the e e o c is expert because it put its entire case or all of its eggs in the basket of trying to demonstrate disparate impact through the use of its expert in this case of course was over the employer's use of credit checks and the district court judge indicated this was something the e its e e o c itself did for eighty four of its seventy seven positions what the district court found was that the e e o c s expert report had no legs and it was somewhat of a moving target and i think the facts of that moving target and the timing of them moving target are important the facts in the record show that the original report that the e e o c s expert tendered was on in may one two thousand and twelve but then it was supplemented on august seventeenth two thousand and twelve and judge us to about the sample size actually that report analyzed one thousand and ninety applications from a vendor called g i s which had a base of forty four thousand six hundred seventy applications but at the same time that august seventeenth report ignored seven hundred nine applications from higher right a second vendor and twelve hundred fifty applications and people soft do you know offhand i mean what were the what was the total number of applications for this for the relevant period i mean so you just said this g i s vendor had a total of forty six seventy arabs is that all of them are in their war we believe that a record would show that with all of them so if the grand total of the universe would be in excess of about six thousand six thousand and seven thousand that's in the record somewhere yes or ten days later the expert dr saad for the defendant tendered his report and then on september five there was a new report from dr murphy and the defendants moved to strike it because it was not a rebuttal court it was our position that it was a new report the district court judge ruled on that on october five the district court said i'm going to allow that report but no more energy order saying that is the end report while dr murphy was deposed on november eighth and by my count for the fourth time he supplement his report published a new report that relied upon an additional eighty applications then december twenty one fifth the report where he submitted a report talking about the fifty seven an cooperating applications so you had a series of reports a moving target and as your questions pointed out it was always the burden of the sea to demonstrate the reliability under the dog standards of the expert report what's your response to miss northrop's point that three of the daubert factors here did have some support that the just accord overlooked i guess the testability standards of controls and i point by point selected those issues in and analyzed whether or not the e e o c has proved what it tender was sufficient to demonstrates those points the e e o c quarrels with the district court and says she aired brief talks about the errors by the district court but for the first time in oral argument this morning the e e o c admits that well this is about an abuse of discretion and whether or not insisting through those factors and analyzing them the judge was so manifestly incorrect or went so beyond the bounds of what the law requires that that somehow constituted an abuse of discretion why don't you think the fifties checking the fifty seven ratings for which murphy had soft identification data wise and that some support for the idea that this could be tested that his rating methodology could be tested because it was done on december twenty one months after the october five order when the judge said no more reports where the defense said pointed out time after time dr murphy's errors this is not the first time dr murphy procedural orders that the court has entered and if we set aside the procedural point that it was untimely and i certainly understand the point you make there what's your response on the substance of their argument that these fifty seven show that it was testable it was not scientific it wasn't done through a random sample the e e o c admitted much in its briefs in response to your question in mid in a footnote in their briefs said it was not a scientific study it wasn't a random sample in the size of the sample wouldn't pass muster under dog standards either and so it was a a snapshot of something that was in response to a critique of the expert report that came so far down the line in the district court certainly didn't abuse it discretion in saying the reasons you've now tendered to demonstrate its reliability don't hold weight v o c tends to find fault with the district court in terms of applying dogbert should it be one of the four that's important all the for this court's decisions in siegler and beyond say it's up to the district court to look at the facts and circumstances but before it and then in a flexible manner apply those dawber to standards that's precisely what the district court judge did in this particular case she obviously knew the law she was adept at the law had been through the process went through each of the factors and found that the report was lacking in reliability could have done many things to prove and demonstrate their reliability of its report we could have conducted telephone interviews of the people who could have filed a motion with the court seeking to send a questionnaire the people where they self identified themselves for whatever reason after four years of an investigation and four years of litigation the e e o c decided to do it through this rating system with the photograph something that other courts and said was inherently unreliable and as they put all their eggs in one basket and their whole case depended upon that expert study that the district court found to be unreliable it makes sense that the district court judge would basically said there's no way in the world that the see can prove disparate impact and summary judgment controversial and this was not something that was accepted in the sign community and so when it comes to the issue of abuse of discretion we believe that the factors that were front of her were the appropriate factors and there's no abuse  northrup thank you briefly to address some of the issues that were raised by kaplan the use of tell of the telephone interview surveys in face to face surveys as an option the e e o c presented undisputed testimony through its expert that the use of paper surveys and questionnaires has an unacceptably low return rate and through a study that he presented showed that telephone interviews tend to over represent whites and that face to face interviews as compared to telephone tend to over represent minorities kaplan did not make its record on this point and it had every opportunity to oppose that evidence the expert explained that that is why he augmented the self identification race data over representation of the sort you just mentioned renders data i guess unreliable for purposes of crowing conclusions what are we to make the overrepresentation fail rates in the in the applications of mr dr moore this is tied up with kaplan's focus on procedure because the alleged disparity and fail rate existed only with regard to the original data sent of ten ninety that was added to once our expert was able to open kathleen's people soft data and add the eight hundred eight persons kaplan credit checked in hired it went down but it didn't go away it went to fifteen point seven percent which is almost the same fail rate as that in the generalized applicant applicant pool doesn't i mean but  percent swing it doesn't give one pause if one thinks about what data was added the data that was added were all passes because they were all from kaplan's higher data set showing people who are credit checked and hired but that one would not  i s data who some sort of credit check was done on it our expert testified that he searched for as many outcomes as he could find pass fail records and started there and he included every single person for whom he found a pass fail record those persons were not immediately drivable from that data set it had to be compiled using three different sources there was no universally coded database here he compiled that and he found all of the persons who could be said to have pas failed gathering that data was as much by chance as everything else in other words if the outcomes were driveable from kaplan's records he included them kaplan has not identified a single person ever whose outcome was known who we left out and they have not identified a single person whose race there is nobody who they sent their job to disprove the conclusions of the was a sample of those whose race and outcomes were known every single person who's race and data could be found was included in this analysis and was analyzed by our expert and i only point out that kaplan has not identified anywhere when we left out because kevin's in the unique position here of being the decision maker this is not like any other kind of data this is data that kaplan had control over at some point they were the ones who made these decisions and the fact that they have not found anyone we left out i think matters because the context matters i'm not suggesting we don't have a burden at all but i'm suggesting is that here we satisfied that burden and the court rejected it based on  be considered it's not part of this methodology was not in the expert did not intend for them to use no it's not as the way of doing it so the names were not to be considered they're not part of the methodology correct it was not correct the expert do not intend for that to be the methodology ok and there's no evidence that they were used although obviously in the brief we take the position that that would not be a terrible point of data to use but yes it was not part of his plan to have them identify people based on last name ok thank you very much